Oct 29 22:13:38.683052 ip-10-0-143-15 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:13:39.021148 ip-10-0-143-15 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:39.021148 ip-10-0-143-15 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:13:39.021738 ip-10-0-143-15 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:39.021738 ip-10-0-143-15 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:39.026241 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026161    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.026241 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026240    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.026241 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026245    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026248    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026251    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026254    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026256    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026259    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026262    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026264    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026267    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026272    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026276    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026279    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026282    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026285    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026288    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026291    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026293    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026296    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026299    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026302    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.026365 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026304    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026307    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026310    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026313    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026316    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026324    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026327    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026329    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026332    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026335    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026338    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026340    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026343    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026345    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026349    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026352    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026355    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026358    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026360    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026363    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.026927 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026366    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026369    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026372    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026374    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026378    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026381    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026383    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026386    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026389    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026391    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026394    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026397    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026399    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026403    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026407    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026411    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026413    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026417    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026419    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.027427 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026422    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026425    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026428    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026430    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026433    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026435    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026438    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026441    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026444    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026447    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026449    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026452    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026455    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026457    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026460    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026462    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026465    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026468    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026470    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.027884 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026473    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026475    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026479    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026482    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026484    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026486    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026489    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026491    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026494    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026497    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026499    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026502    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026505    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026508    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026511    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026513    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026516    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026518    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026520    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.028338 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026523    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026527    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026529    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026532    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026535    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026538    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.026540    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028130    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028138    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028141    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028144    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028148    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028151    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028154    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028156    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028159    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028162    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028164    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028167    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.028794 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028169    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028172    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028175    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028178    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028181    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028183    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028186    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028189    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028193    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028197    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028201    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028203    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028207    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028210    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028213    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028216    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028218    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028221    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028224    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.029261 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028227    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028229    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028232    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028234    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028237    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028239    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028242    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028244    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028247    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028249    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028252    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028254    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028257    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028259    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028262    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028265    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028267    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028270    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028272    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028275    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.029710 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028278    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028281    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028283    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028286    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028288    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028291    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028293    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028296    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028299    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028301    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028304    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028307    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028309    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028313    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028317    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028320    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028322    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028325    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028328    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.030216 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028330    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028333    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028335    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028338    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028341    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028344    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028346    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028349    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028352    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028354    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028357    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028359    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028362    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028365    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028367    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028370    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028373    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028376    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028378    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028381    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.030659 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028383    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028386    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028388    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028391    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028394    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028396    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028398    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028401    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028403    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028406    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028408    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028411    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028413    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028417    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028419    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028422    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028952    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028957    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028960    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028963    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.031144 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028966    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028968    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028971    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028973    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028976    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028979    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028981    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028984    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028987    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028990    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028992    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028995    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.028998    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029000    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029003    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029007    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029010    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029014    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029017    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.031608 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029020    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029022    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029024    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029027    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029029    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029032    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029034    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029037    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029040    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029042    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029045    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029048    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029058    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029061    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029064    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029067    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029069    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029072    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029075    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029077    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.032058 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029080    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029083    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029086    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029089    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029091    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029094    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029097    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029099    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029102    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029104    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029107    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029110    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029113    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029116    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029118    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029121    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029123    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029126    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029128    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029131    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.032518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029134    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029136    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029139    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029141    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029144    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029146    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029149    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029152    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029155    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029160    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029162    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029165    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029167    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029170    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029173    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029175    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029178    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029183    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029186    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.032993 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029188    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029191    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029193    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029196    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029199    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029201    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029204    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029206    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029209    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029211    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029214    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029216    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029218    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029221    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029224    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029226    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029229    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029231    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029234    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.033453 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029236    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029239    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029241    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029244    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.029246    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030065    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030074    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030080    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030085    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030090    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030094    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030100    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030105    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030108    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030111    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030115    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030118    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030121    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030124    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030127    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030130    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030133    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030136    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030140    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:13:39.033921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030143    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030146    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030151    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030155    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030159    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030162    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030165    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030168    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030171    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030174    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030177    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030180    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030183    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030187    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030191    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030193    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030197    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030199    2578 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030203    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030209    2578 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030212    2578 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030215    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030219    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030222    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:13:39.034477 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030227    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030230    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030233    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030236    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030239    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030242    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030245    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030248    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030251    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030254    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030259    2578 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030263    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030266    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030269    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030273    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030276    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030279    2578 flags.go:64] FLAG: --help="false"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030282    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030285    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030288    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030291    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030294    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030297    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030301    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:13:39.035116 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030304    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030307    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030310    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030314    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030317    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030320    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030323    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030326    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030329    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030332    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030335    2578 flags.go:64] FLAG: --lock-file=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030338    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030341    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030344    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030349    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030352    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030355    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030358    2578 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030363    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030366    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030369    2578 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030372    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030376    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030380    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030383    2578 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:13:39.035679 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030386    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030389    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030393    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030396    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030399    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030402    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030405    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030412    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030415    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030418    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030421    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030424    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030427    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030430    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030433    2578 flags.go:64] FLAG: --port="10250"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030436    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030439    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f2e8fdfa2b152b75"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030442    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030445    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030448    2578 flags.go:64] FLAG: --register-node="true"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030451    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030455    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030458    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030461    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030464    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030471    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:13:39.036294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030474    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030477    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030480    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030483    2578 flags.go:64] FLAG: --runonce="false"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030486    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030489    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030492    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030495    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030498    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030501    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030504    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030507    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030511    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030514    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030517    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030520    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030524    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030527    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030530    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030535    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030538    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030541    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030546    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030548    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030551    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:13:39.036921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030554    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030557    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030560    2578 flags.go:64] FLAG: --v="2"
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030565    2578 flags.go:64] FLAG: --version="false"
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030568    2578 flags.go:64] FLAG: --vmodule=""
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030573    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.030577    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030726    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030730    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030733    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030736    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030739    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030742    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030745    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030747    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030751    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030754    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030758    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030761    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030763    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030766    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.037518 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030768    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030771    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030774    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030790    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030793    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030796    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030798    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030801    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030804    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030807    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030809    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030812    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030814    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030817    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030820    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030822    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030825    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030829    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030832    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.038041 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030835    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030838    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030840    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030843    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030845    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030848    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030851    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030854    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030857    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030861    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030865    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030868    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030871    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030873    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030876    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030880    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030883    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030886    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030888    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030891    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.038499 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030894    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030896    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030899    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030901    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030904    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030907    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030909    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030912    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030914    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030917    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030921    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030923    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030926    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030929    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030931    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030934    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030936    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030939    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030942    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.039057 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030945    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030947    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030949    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030952    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030955    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030957    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030960    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030962    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030966    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030969    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030971    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030974    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030976    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030979    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030982    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030985    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030988    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030990    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030992    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.039514 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030995    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.030998    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031000    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031003    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031007    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031010    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031013    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031015    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031018    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031020    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031023    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031026    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031028    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031031    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031034    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.039977 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.031060    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031093    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031097    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031100    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031103    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031106    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031110    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031112    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031115    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031117    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031120    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031123    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031126    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031129    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031131    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031134    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031137    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031139    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031142    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.040333 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031144    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031147    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031150    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031155    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031157    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031160    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031163    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031165    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031168    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031171    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031173    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031176    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031179    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031181    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031184    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031186    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031189    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031193    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031197    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031201    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.040771 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031203    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031206    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031209    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031211    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031214    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031216    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031220    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031222    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031225    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031228    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031230    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031233    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031235    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031238    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031240    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031244    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031246    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031249    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031251    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.041258 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031254    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031256    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031259    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031261    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031264    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031267    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031269    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031272    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031275    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031277    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031280    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031283    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031288    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031291    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031294    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031297    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031299    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031302    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031304    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.041699 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031307    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031310    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031312    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031315    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031317    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031320    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031322    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031325    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031327    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031330    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031332    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031335    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031337    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031340    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031342    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031345    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031347    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031350    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031353    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031355    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.042174 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031357    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031360    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031362    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031365    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031367    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031371    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031374    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031376    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031379    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.031382    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.037794    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.037810    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.038412    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.038419    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038474    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038479    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038482    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038486    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038488    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.042700 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038491    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038495    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038499    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038503    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038506    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038509    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038512    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038515    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038518    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038521    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038523    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038526    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038529    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038533    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038536    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038539    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038541    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038544    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038547    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.043176 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038549    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038552    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038555    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038557    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038560    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038562    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038565    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038568    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038570    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038573    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038576    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038578    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038581    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038583    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038586    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038588    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038591    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038593    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038596    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038598    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.043616 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038601    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038604    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038606    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038609    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038611    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038614    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038616    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038619    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038621    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038624    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038627    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038629    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038632    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038635    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038637    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038640    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038642    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038645    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038647    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.044139 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038650    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038652    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038655    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038658    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038660    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038663    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038665    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038668    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038670    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038673    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038675    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038678    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038680    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038683    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038685    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038688    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038690    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038693    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038695    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.044601 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038698    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038700    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038703    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038705    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038708    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038711    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038713    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038716    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038719    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038721    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038724    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038726    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038729    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038732    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038734    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038737    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038739    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038742    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038744    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038747    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.045346 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038749    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038752    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038754    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038757    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.038800    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038954    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038959    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038962    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038965    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038967    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038970    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038972    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038975    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038978    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038980    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038983    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.046130 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038986    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038988    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038991    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038994    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038996    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.038999    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039001    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039004    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039006    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039009    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039011    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039014    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039016    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039019    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039022    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039024    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039027    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039029    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039032    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.046757 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039034    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039037    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039040    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039042    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039045    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039047    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039049    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039052    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039055    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039057    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039060    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039063    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039065    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039068    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039071    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039074    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039076    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039079    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039081    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039084    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.047631 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039086    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039089    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039091    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039094    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039096    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039099    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039101    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039104    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039107    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039109    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039112    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039114    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039117    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039119    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039122    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039124    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039127    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039129    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039132    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039134    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.048159 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039137    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039140    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039144    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039148    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039151    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039153    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039158    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039161    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039164    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039166    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039169    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039171    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039174    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039176    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039179    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039181    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039184    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039186    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039189    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.048676 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039192    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039194    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039197    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039200    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039202    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039205    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039207    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039210    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039212    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039215    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039232    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039236    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039240    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039243    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039246    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039249    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.049281 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039251    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.039275    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039305    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039311    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039317    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039320    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039323    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039327    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039330    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039334    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039338    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039341    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039343    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039346    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039348    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039351    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:39.049648 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039354    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039356    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039359    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039362    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039364    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039367    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039370    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039372    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039375    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039377    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039380    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039382    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039385    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039388    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039390    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039393    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039396    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039398    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039400    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:39.050121 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039404    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039409    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039411    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039414    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039417    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039419    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039422    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039425    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039427    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039430    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039432    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039435    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039438    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039440    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039443    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039445    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039449    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039451    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039454    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039457    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:39.050606 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039459    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039462    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039464    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039467    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039470    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039472    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039475    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039477    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039480    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039482    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039485    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039488    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039492    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039496    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039499    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039501    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039504    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039506    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039509    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039512    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:39.051105 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039514    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039517    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039520    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039522    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039525    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039528    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039530    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039533    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039536    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039539    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039542    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039546    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039549    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039552    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039555    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039558    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039560    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039563    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039566    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:39.051581 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039568    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039571    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039573    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039576    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039578    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039582    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039586    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039588    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039591    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039594    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039597    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039599    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039602    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:39.039604    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.039650    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.042304    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.043073    2578 server.go:1014] "Starting client certificate rotation"
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.043164    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:13:39.052188 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.043193    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:13:39.052632 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.045533    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:13:39.052632 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.047657    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:13:39.056394 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.056381    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:13:39.060854 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.060836    2578 log.go:25] "Validated CRI v1 image API"
Oct 29 22:13:39.062488 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.062463    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:13:39.064692 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.064671    2578 fs.go:135] Filesystem UUIDs: map[5847d7cd-75dc-450c-b0fa-a8ba71e0f63a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cbd286bb-b1a4-4625-8f14-6f34f2f81848:/dev/nvme0n1p3]
Oct 29 22:13:39.064797 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.064690    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:13:39.069597 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.069473    2578 manager.go:217] Machine: {Timestamp:2026-10-29 22:13:39.068326946 +0000 UTC m=+0.285513016 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100259 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2218f61520bd412796ea8c12327573 SystemUUID:ec2218f6-1520-bd41-2796-ea8c12327573 BootID:fe01e702-c8d7-41e3-8fc8-6fac3f6585b6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:e7:b1:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:e7:b1:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:e6:94:39:9e:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:13:39.069597 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.069594    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:13:39.069752 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.069694    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:13:39.072292 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.072265    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:13:39.072424 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.072292    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:13:39.072466 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.072426    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:13:39.072466 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.072434    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:13:39.072466 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.072458    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:13:39.072546 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.072470    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:13:39.073327 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.073318    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:13:39.077164 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077152    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:13:39.077280 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077271    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:13:39.077921 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077911    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:13:39.077959 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077925    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:13:39.077959 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077939    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:13:39.077959 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077949    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:13:39.077959 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.077956    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:13:39.078267 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.078254    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:13:39.080023 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.080009    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:13:39.086766 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.086751    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:13:39.087494 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087482    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:13:39.087530 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087512    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:13:39.087656 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087646    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:13:39.087694 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087661    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:13:39.087694 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087667    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:13:39.087694 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087672    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:13:39.087694 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087678    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:13:39.087694 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.087683    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:13:39.088211 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088202    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:13:39.088280 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088212    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:13:39.088280 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088219    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:13:39.088280 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088225    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:13:39.088280 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088233    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:13:39.088280 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088242    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:13:39.088893 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088883    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:13:39.088893 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.088893    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:13:39.092377 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.092362    2578 server.go:1263] "Started kubelet"
Oct 29 22:13:39.092479 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.092454    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:13:39.092533 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.092459    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:13:39.092579 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.092530    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:13:39.093187 ip-10-0-143-15 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:13:39.094194 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.093730    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:13:39.094820 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.094695    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:13:39.098198 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.098172    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:13:39.099788 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.099762    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:13:39.100694 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.100681    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:13:39.100830 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.100806    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.101299 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101280    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:13:39.101391 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101284    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:13:39.101391 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101316    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:13:39.101464 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101410    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:13:39.101464 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101417    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:13:39.101519 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101511    2578 factory.go:55] Registering systemd factory
Oct 29 22:13:39.101546 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101542    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:13:39.101726 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101715    2578 factory.go:153] Registering CRI-O factory
Oct 29 22:13:39.101768 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101729    2578 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:13:39.101821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101771    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:13:39.101821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101803    2578 factory.go:103] Registering Raw factory
Oct 29 22:13:39.101821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.101818    2578 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:13:39.102837 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.102822    2578 manager.go:319] Starting recovery of all containers
Oct 29 22:13:39.114627 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.114605    2578 manager.go:324] Recovery completed
Oct 29 22:13:39.118894 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.118883    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.120549 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.120532    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.120623 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.120559    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.120623 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.120569    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.121135 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121121    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:13:39.121135 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121133    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:13:39.121253 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121150    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:13:39.121253 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121196    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.121911 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121894    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.121965 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121924    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.121965 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.121938    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.123433 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.123419    2578 policy_none.go:50] "Start"
Oct 29 22:13:39.123493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.123437    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:13:39.123493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.123447    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:13:39.125506 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.125493    2578 policy_none.go:44] "Start"
Oct 29 22:13:39.129481 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.129448    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:13:39.129607 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.129473    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:13:39.130415 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.129457    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-15.ec2.internal.18e31fc7ad6ff1cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-15.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-15.ec2.internal,},FirstTimestamp:2026-10-29 22:13:39.092337099 +0000 UTC m=+0.309523180,LastTimestamp:2026-10-29 22:13:39.092337099 +0000 UTC m=+0.309523180,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-15.ec2.internal,}"
Oct 29 22:13:39.131506 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.131488    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:13:39.131624 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.131611    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmk44"
Oct 29 22:13:39.131688 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.131632    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmk44"
Oct 29 22:13:39.143911 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.143872    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:13:39.143992 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.143925    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 22:13:39.143992 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.143955    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 22:13:39.143992 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.143898    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-15.ec2.internal.18e31fc7af1e5fdb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-15.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-15.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-15.ec2.internal,},FirstTimestamp:2026-10-29 22:13:39.120545755 +0000 UTC m=+0.337731824,LastTimestamp:2026-10-29 22:13:39.120545755 +0000 UTC m=+0.337731824,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-15.ec2.internal,}"
Oct 29 22:13:39.165361 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.165328    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:13:39.167705 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.167689    2578 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:13:39.168291 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.168269    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:13:39.168291 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.168292    2578 server.go:87] "Starting device plugin registration server"
Oct 29 22:13:39.168535 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.168521    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:13:39.168594 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.168535    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:13:39.168648 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.168626    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:13:39.168973 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.168693    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:13:39.168973 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.168701    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:13:39.169194 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.169174    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:13:39.169257 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.169223    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.240270 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.240251    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:13:39.241436 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.241415    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:13:39.241436 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.241437    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:13:39.241568 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.241456    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:13:39.241568 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.241493    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 22:13:39.256775 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.256750    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:39.268821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.268808    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.269645 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.269622    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.269721 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.269652    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.269721 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.269663    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.269721 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.269682    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.286526 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.286478    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.286526 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.286500    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-15.ec2.internal\": node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.322896 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.322879    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.341636 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.341593    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"]
Oct 29 22:13:39.341705 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.341664    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.342427 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.342413    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.342505 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.342437    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.342505 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.342447    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.344735 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.344722    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.344883 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.344868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.344932 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.344898    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.345540 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.345494    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.345540 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.345518    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.345540 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.345527    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.345540 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.345537    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.345703 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.345556    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.345703 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.345570    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.347646 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.347633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.347704 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.347656    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:39.348283 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.348268    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:39.348347 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.348292    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:39.348347 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.348302    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:39.376299 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.376279    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-15.ec2.internal\" not found" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.379139 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.379123    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-15.ec2.internal\" not found" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.402562 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.402502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.402625 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.402571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.402625 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.402589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b64de3cb655c6c434e0a94257c87f3e3-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"b64de3cb655c6c434e0a94257c87f3e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.423923 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.423904    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.503645 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.503626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.503726 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.503647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.503726 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.503700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.503726 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.503717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b64de3cb655c6c434e0a94257c87f3e3-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"b64de3cb655c6c434e0a94257c87f3e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.503862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.503721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.503862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.503743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b64de3cb655c6c434e0a94257c87f3e3-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"b64de3cb655c6c434e0a94257c87f3e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.524567 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.524549    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.625075 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.625030    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.680246 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.680212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.682362 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:39.682342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:39.726014 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.725876    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.826456 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.826430    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:39.926904 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:39.926845    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:40.027371 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:40.027350    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:40.043528 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:40.043504    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:13:40.043640 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:40.043623    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:13:41.350523 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:40.127457    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:41.355392 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.355357    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:08:40 +0000 UTC" deadline="2028-05-12 22:31:23.35521364 +0000 UTC"
Oct 29 22:13:41.355392 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.355390    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13464h17m41.999827203s"
Oct 29 22:13:41.443251 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.443196    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:41.444358 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.444335    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:41.444476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.444366    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:41.444476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.444375    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:41.447520 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:41.447505    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:41.472327 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.472308    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:41.509646 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:41.509604    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64de3cb655c6c434e0a94257c87f3e3.slice/crio-f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616 WatchSource:0}: Error finding container f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616: Status 404 returned error can't find the container with id f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616
Oct 29 22:13:41.509816 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:41.509801    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06e61f905fbd53de8ba68af88289e72.slice/crio-6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170 WatchSource:0}: Error finding container 6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170: Status 404 returned error can't find the container with id 6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170
Oct 29 22:13:41.514978 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.514963    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:13:41.548464 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:41.548447    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:41.648757 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:41.648732    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:13:41.663772 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.663751    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:41.664003 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.663990    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:41.720022 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.719968    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:41.743142 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.743128    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:41.744119 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.744104    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:13:41.763623 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:41.763609    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:42.298314 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.298283    2578 apiserver.go:52] "Watching apiserver"
Oct 29 22:13:42.319445 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.319417    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:13:42.337801 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.337776    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:13:42.346732 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.346055    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-8mh74","kube-system/konnectivity-agent-x4bzm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x","openshift-dns/node-resolver-l7cws","openshift-multus/multus-ngnk6","openshift-network-diagnostics/network-check-target-qp2mt","openshift-network-operator/iptables-alerter-wx7df","kube-system/global-pull-secret-syncer-j9lz9","kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7nhrb","openshift-image-registry/node-ca-5q62r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal","openshift-multus/network-metrics-daemon-g7bbj","openshift-multus/multus-additional-cni-plugins-hnw4v"]
Oct 29 22:13:42.348865 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.348841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.352451 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.352369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:42.352451 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.352427    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:42.354554 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.354531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.357168 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.357147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.359476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.359458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.361501 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.361475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:42.361600 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.361559    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:42.363531 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.363469    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.363531 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.363470    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.363531 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.363509    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:13:42.363723 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.363574    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f584j\""
Oct 29 22:13:42.366222 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.365954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.368043 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.367950    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:13:42.368150 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.368054    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gv8cl"
Oct 29 22:13:42.368396 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.368380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.371237 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.370740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.373690 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.373095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.376432 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.375466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.376432 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.376174    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gv8cl"
Oct 29 22:13:42.377903 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.377859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.377991 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.377975    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:42.421416 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.421396    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:13:42.434596 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-netns\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.434692 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-config\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.434692 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-kubernetes\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.434776 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-serviceca\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.434810 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.434846 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwl6\" (UniqueName: \"kubernetes.io/projected/f87a90af-8294-4303-b000-6b4e52289f79-kube-api-access-4kwl6\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.434879 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.434906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-device-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r8z\" (UniqueName: \"kubernetes.io/projected/0c6470dd-33de-42f4-a144-2fe004a117be-kube-api-access-88r8z\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-netns\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-slash\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-netd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdhq\" (UniqueName: \"kubernetes.io/projected/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-kube-api-access-5hdhq\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-dbus\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-system-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-run\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-etc-tuned\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmb7h\" (UniqueName: \"kubernetes.io/projected/95a82d11-1674-4b89-bdeb-6fbf6caa5304-kube-api-access-lmb7h\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-k8s-cni-cncf-io\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-etc-kubernetes\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.435629 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-ovn\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnh8\" (UniqueName: \"kubernetes.io/projected/7520d944-da53-4455-868f-3880e5535345-kube-api-access-5pnh8\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-systemd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-etc-selinux\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqvn\" (UniqueName: \"kubernetes.io/projected/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-kube-api-access-tvqvn\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-socket-dir-parent\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-kubelet\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-hostroot\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-multus-daemon-config\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-hosts-file\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-cnibin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-multus-certs\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-bin\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7bd\" (UniqueName: \"kubernetes.io/projected/26757508-3500-4597-b243-ae17376bd037-kube-api-access-bt7bd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.436426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-systemd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.435977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7520d944-da53-4455-868f-3880e5535345-ovn-node-metrics-cert\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-sys\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-tmp\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-system-cni-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-var-lib-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-etc-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysconfig\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-conf\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-var-lib-kubelet\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-host\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-kubelet-config\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.437183 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-tmp-dir\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-log-socket\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-script-lib\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e49e136-156b-4dfa-a877-51b5c57d0ffa-agent-certs\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-modprobe-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-socket-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-conf-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-lib-modules\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-sys-fs\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-host\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95a82d11-1674-4b89-bdeb-6fbf6caa5304-iptables-alerter-script\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-env-overrides\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-registration-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.437995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-cnibin\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-node-log\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbtf\" (UniqueName: \"kubernetes.io/projected/30bf6468-e2cc-4f55-a719-c33ab521bf43-kube-api-access-ksbtf\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95a82d11-1674-4b89-bdeb-6fbf6caa5304-host-slash\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-cni-binary-copy\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-bin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.436978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-systemd-units\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.437003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e49e136-156b-4dfa-a877-51b5c57d0ffa-konnectivity-ca\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.437030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzhb\" (UniqueName: \"kubernetes.io/projected/e36a49f7-9cf1-4bd1-aed4-c66384880090-kube-api-access-nzzhb\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.437054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-multus\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.438456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.437079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-kubelet\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.466069 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466017    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"e06e61f905fbd53de8ba68af88289e72","Type":"ContainerStarted","Data":"6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170"}
Oct 29 22:13:42.466289 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466267    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:13:42.466289 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466268    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.466692 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466674    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:13:42.466757 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466734    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:13:42.466806 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466755    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:13:42.466857 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466820    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:13:42.466907 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.466679    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.467060 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467042    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr68l\""
Oct 29 22:13:42.467172 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467154    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p5dkc\""
Oct 29 22:13:42.467172 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467166    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwjl8\""
Oct 29 22:13:42.467294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467228    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:13:42.467294 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467275    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.467360 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467292    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:13:42.467360 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467323    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.467360 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467343    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:13:42.467360 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467356    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467276    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467381    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwm9d\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467411    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467416    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467429    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467447    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjh7c\""
Oct 29 22:13:42.467476 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.467454    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.468071 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.468049    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" event={"ID":"b64de3cb655c6c434e0a94257c87f3e3","Type":"ContainerStarted","Data":"f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616"}
Oct 29 22:13:42.478949 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.478929    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.478949 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.478944    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:13:42.479110 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.478963    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.479110 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.479003    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fz5pc\""
Oct 29 22:13:42.479110 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.479022    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cdnlg\""
Oct 29 22:13:42.479110 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.479054    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:13:42.479110 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.479057    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjwp\""
Oct 29 22:13:42.479110 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.479061    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:13:42.479408 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.479392    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.537414 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-device-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.537585 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r8z\" (UniqueName: \"kubernetes.io/projected/0c6470dd-33de-42f4-a144-2fe004a117be-kube-api-access-88r8z\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.537585 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-netns\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.537585 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-netns\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.537585 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-device-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.537585 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-slash\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-netd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdhq\" (UniqueName: \"kubernetes.io/projected/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-kube-api-access-5hdhq\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-dbus\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-system-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-run\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-etc-tuned\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmb7h\" (UniqueName: \"kubernetes.io/projected/95a82d11-1674-4b89-bdeb-6fbf6caa5304-kube-api-access-lmb7h\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.537809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-k8s-cni-cncf-io\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-etc-kubernetes\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-ovn\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnh8\" (UniqueName: \"kubernetes.io/projected/7520d944-da53-4455-868f-3880e5535345-kube-api-access-5pnh8\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-systemd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-etc-selinux\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqvn\" (UniqueName: \"kubernetes.io/projected/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-kube-api-access-tvqvn\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537932    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-socket-dir-parent\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-kubelet\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537991    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-netd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.537995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-hostroot\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-multus-daemon-config\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-hostroot\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.538602 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538215    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-dbus\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.538602 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538286    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.538602 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-system-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538602 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-run\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.538602 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-multus-daemon-config\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538602 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538564    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-systemd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.538764 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538681    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-etc-kubernetes\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538764 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538731    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-ovn\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.538764 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538751    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538846 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538794    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-k8s-cni-cncf-io\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538846 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538664    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:13:42.538899 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538851    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-kubelet\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538899 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538889    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-socket-dir-parent\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.538981 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538932    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-slash\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.538981 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-hosts-file\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.539078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.538994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-cnibin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.539078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539009    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-etc-selinux\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.539078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-multus-certs\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.539078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:42.539078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-hosts-file\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.539078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-bin\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-multus-certs\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7bd\" (UniqueName: \"kubernetes.io/projected/26757508-3500-4597-b243-ae17376bd037-kube-api-access-bt7bd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-cnibin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-systemd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7520d944-da53-4455-868f-3880e5535345-ovn-node-metrics-cert\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-bin\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-sys\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-tmp\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-system-cni-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-var-lib-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-etc-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539329 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysconfig\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-conf\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-var-lib-kubelet\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-host\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-kubelet-config\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-tmp-dir\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-log-socket\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-script-lib\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e49e136-156b-4dfa-a877-51b5c57d0ffa-agent-certs\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-modprobe-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-socket-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-conf-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-lib-modules\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.539862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-sys-fs\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-host\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95a82d11-1674-4b89-bdeb-6fbf6caa5304-iptables-alerter-script\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-env-overrides\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-registration-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-cnibin\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-node-log\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbtf\" (UniqueName: \"kubernetes.io/projected/30bf6468-e2cc-4f55-a719-c33ab521bf43-kube-api-access-ksbtf\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95a82d11-1674-4b89-bdeb-6fbf6caa5304-host-slash\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-cni-binary-copy\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-bin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-systemd-units\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e49e136-156b-4dfa-a877-51b5c57d0ffa-konnectivity-ca\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzhb\" (UniqueName: \"kubernetes.io/projected/e36a49f7-9cf1-4bd1-aed4-c66384880090-kube-api-access-nzzhb\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-multus\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-kubelet\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.540493 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-netns\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-config\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-kubernetes\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-serviceca\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwl6\" (UniqueName: \"kubernetes.io/projected/f87a90af-8294-4303-b000-6b4e52289f79-kube-api-access-4kwl6\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540303    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-lib-modules\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540357    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-system-cni-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540413    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-sys-fs\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-host\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540499    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.540636    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.540704    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:43.040682872 +0000 UTC m=+3.038737855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-sys\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.541276 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.540990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95a82d11-1674-4b89-bdeb-6fbf6caa5304-iptables-alerter-script\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541176    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-tmp-dir\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541258    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-log-socket\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541303    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-var-lib-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-etc-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541399    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysconfig\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541445    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-env-overrides\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541496    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-script-lib\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541504    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-conf\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541536    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-registration-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541551    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-var-lib-kubelet\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-cnibin\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541601    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-host\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541626    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-node-log\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95a82d11-1674-4b89-bdeb-6fbf6caa5304-host-slash\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541884    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-kubelet\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542078 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.541956    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-netns\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.542254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-kubernetes\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.542821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.539623    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-systemd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.542821 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.542340    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-modprobe-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.542993    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.543060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret podName:bbeed871-241a-4509-ac94-dbe5170904fe nodeName:}" failed. No retries permitted until 2026-10-29 22:13:43.043042453 +0000 UTC m=+3.041097421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret") pod "global-pull-secret-syncer-j9lz9" (UID: "bbeed871-241a-4509-ac94-dbe5170904fe") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543060    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-etc-tuned\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-bin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543227    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-kubelet-config\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543313    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-config\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-multus\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543406    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-systemd-units\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543417    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543429    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-conf-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543644    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e49e136-156b-4dfa-a877-51b5c57d0ffa-konnectivity-ca\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.543684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-tmp\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.544048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-serviceca\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.544056    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-cni-binary-copy\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.544151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.544134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-socket-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.545285 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.545262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e49e136-156b-4dfa-a877-51b5c57d0ffa-agent-certs\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.545422 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.545384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.548573 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.548412    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7520d944-da53-4455-868f-3880e5535345-ovn-node-metrics-cert\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.557571 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.557476    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:42.557571 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.557493    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:42.557571 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.557502    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zd9bm for pod openshift-network-diagnostics/network-check-target-qp2mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:42.557571 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:42.557547    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm podName:ffa11b25-01d3-4d6a-ad70-fcfe34ad324c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:43.057533697 +0000 UTC m=+3.055588663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zd9bm" (UniqueName: "kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm") pod "network-check-target-qp2mt" (UID: "ffa11b25-01d3-4d6a-ad70-fcfe34ad324c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:42.558169 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.558120    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmb7h\" (UniqueName: \"kubernetes.io/projected/95a82d11-1674-4b89-bdeb-6fbf6caa5304-kube-api-access-lmb7h\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.559411 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.559389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnh8\" (UniqueName: \"kubernetes.io/projected/7520d944-da53-4455-868f-3880e5535345-kube-api-access-5pnh8\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.562933 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.560309    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdhq\" (UniqueName: \"kubernetes.io/projected/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-kube-api-access-5hdhq\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.562933 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.560334    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r8z\" (UniqueName: \"kubernetes.io/projected/0c6470dd-33de-42f4-a144-2fe004a117be-kube-api-access-88r8z\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.562933 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.560468    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7bd\" (UniqueName: \"kubernetes.io/projected/26757508-3500-4597-b243-ae17376bd037-kube-api-access-bt7bd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.562933 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.561308    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbtf\" (UniqueName: \"kubernetes.io/projected/30bf6468-e2cc-4f55-a719-c33ab521bf43-kube-api-access-ksbtf\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.563580 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.563533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqvn\" (UniqueName: \"kubernetes.io/projected/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-kube-api-access-tvqvn\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.565230 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.565180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzhb\" (UniqueName: \"kubernetes.io/projected/e36a49f7-9cf1-4bd1-aed4-c66384880090-kube-api-access-nzzhb\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:42.565505 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.565476    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwl6\" (UniqueName: \"kubernetes.io/projected/f87a90af-8294-4303-b000-6b4e52289f79-kube-api-access-4kwl6\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.623528 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.623500    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:42.664398 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.664368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:13:42.675436 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.675409    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a82d11_1674_4b89_bdeb_6fbf6caa5304.slice/crio-1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b WatchSource:0}: Error finding container 1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b: Status 404 returned error can't find the container with id 1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b
Oct 29 22:13:42.676465 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.676446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:13:42.682456 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.682434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:13:42.687245 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.687220    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6470dd_33de_42f4_a144_2fe004a117be.slice/crio-8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09 WatchSource:0}: Error finding container 8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09: Status 404 returned error can't find the container with id 8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09
Oct 29 22:13:42.689198 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.689172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngnk6"
Oct 29 22:13:42.699066 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.699039    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bc42bb_f14c_4921_a5f4_5a7ceec653e6.slice/crio-17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86 WatchSource:0}: Error finding container 17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86: Status 404 returned error can't find the container with id 17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86
Oct 29 22:13:42.701899 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.701876    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87a90af_8294_4303_b000_6b4e52289f79.slice/crio-6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb WatchSource:0}: Error finding container 6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb: Status 404 returned error can't find the container with id 6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb
Oct 29 22:13:42.713462 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.713307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:13:42.715216 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.715195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:42.724960 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.724934    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bf6468_e2cc_4f55_a719_c33ab521bf43.slice/crio-5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9 WatchSource:0}: Error finding container 5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9: Status 404 returned error can't find the container with id 5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9
Oct 29 22:13:42.728228 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.728209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:13:42.732045 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.732030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:13:42.734144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:42.734123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:13:42.735434 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.735415    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e49e136_156b_4dfa_a877_51b5c57d0ffa.slice/crio-0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766 WatchSource:0}: Error finding container 0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766: Status 404 returned error can't find the container with id 0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766
Oct 29 22:13:42.738532 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.738489    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26757508_3500_4597_b243_ae17376bd037.slice/crio-b84f88fd527e7484a2cb1a8272a17f7e8d22dea7f34e66590e3c6fbc605a0f8e WatchSource:0}: Error finding container b84f88fd527e7484a2cb1a8272a17f7e8d22dea7f34e66590e3c6fbc605a0f8e: Status 404 returned error can't find the container with id b84f88fd527e7484a2cb1a8272a17f7e8d22dea7f34e66590e3c6fbc605a0f8e
Oct 29 22:13:42.744454 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.744436    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520d944_da53_4455_868f_3880e5535345.slice/crio-40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad WatchSource:0}: Error finding container 40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad: Status 404 returned error can't find the container with id 40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad
Oct 29 22:13:42.749068 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:13:42.749051    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0160f0a0_c1cf_4e77_9ed6_4d2cfd45e054.slice/crio-49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e WatchSource:0}: Error finding container 49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e: Status 404 returned error can't find the container with id 49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e
Oct 29 22:13:43.044569 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.044540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:43.044690 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.044612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:43.044751 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.044738    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:43.044807 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.044796    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:44.04477669 +0000 UTC m=+4.042831679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:43.044950 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.044869    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:43.044950 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.044930    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret podName:bbeed871-241a-4509-ac94-dbe5170904fe nodeName:}" failed. No retries permitted until 2026-10-29 22:13:44.044902527 +0000 UTC m=+4.042957511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret") pod "global-pull-secret-syncer-j9lz9" (UID: "bbeed871-241a-4509-ac94-dbe5170904fe") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:43.145234 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.145203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:43.145381 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.145357    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:43.145381 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.145376    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:43.145500 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.145389    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zd9bm for pod openshift-network-diagnostics/network-check-target-qp2mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:43.145500 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.145430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm podName:ffa11b25-01d3-4d6a-ad70-fcfe34ad324c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:44.145419103 +0000 UTC m=+4.143474071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zd9bm" (UniqueName: "kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm") pod "network-check-target-qp2mt" (UID: "ffa11b25-01d3-4d6a-ad70-fcfe34ad324c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:43.379710 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.379552    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:42 +0000 UTC" deadline="2028-04-29 02:22:06.776953423 +0000 UTC"
Oct 29 22:13:43.379710 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.379608    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13132h8m23.397349037s"
Oct 29 22:13:43.462586 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.461702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:43.462586 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.462050    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:43.462586 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.462433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:43.462586 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:43.462531    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:43.512034 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.507954    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x" event={"ID":"0c6470dd-33de-42f4-a144-2fe004a117be","Type":"ContainerStarted","Data":"8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09"}
Oct 29 22:13:43.531581 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.531545    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wx7df" event={"ID":"95a82d11-1674-4b89-bdeb-6fbf6caa5304","Type":"ContainerStarted","Data":"1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b"}
Oct 29 22:13:43.536848 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.536793    2578 generic.go:356] "Generic (PLEG): container finished" podID="e06e61f905fbd53de8ba68af88289e72" containerID="dd523e9c2b668fe002fd40ad34b03e5cc284a95ad664e6d6e709a4aa2a5c04e9" exitCode=0
Oct 29 22:13:43.536982 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.536870    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"e06e61f905fbd53de8ba68af88289e72","Type":"ContainerDied","Data":"dd523e9c2b668fe002fd40ad34b03e5cc284a95ad664e6d6e709a4aa2a5c04e9"}
Oct 29 22:13:43.560463 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.560384    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" event={"ID":"b64de3cb655c6c434e0a94257c87f3e3","Type":"ContainerStarted","Data":"15d0b2c5023eb2b29fe32bbd41dc3a83c45866f0b399f6c3e80bbb0e72d4e70d"}
Oct 29 22:13:43.568381 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.568355    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5q62r" event={"ID":"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054","Type":"ContainerStarted","Data":"49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e"}
Oct 29 22:13:43.572692 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.572666    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad"}
Oct 29 22:13:43.587061 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.587035    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7nhrb" event={"ID":"26757508-3500-4597-b243-ae17376bd037","Type":"ContainerStarted","Data":"b84f88fd527e7484a2cb1a8272a17f7e8d22dea7f34e66590e3c6fbc605a0f8e"}
Oct 29 22:13:43.600158 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.600132    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x4bzm" event={"ID":"3e49e136-156b-4dfa-a877-51b5c57d0ffa","Type":"ContainerStarted","Data":"0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766"}
Oct 29 22:13:43.609646 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.609398    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerStarted","Data":"5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9"}
Oct 29 22:13:43.622426 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.622339    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngnk6" event={"ID":"f87a90af-8294-4303-b000-6b4e52289f79","Type":"ContainerStarted","Data":"6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb"}
Oct 29 22:13:43.625290 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:43.625221    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7cws" event={"ID":"41bc42bb-f14c-4921-a5f4-5a7ceec653e6","Type":"ContainerStarted","Data":"17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86"}
Oct 29 22:13:44.051743 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.051668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:44.051743 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.051735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:44.051976 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.051875    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:44.051976 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.051949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret podName:bbeed871-241a-4509-ac94-dbe5170904fe nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.051929935 +0000 UTC m=+6.049984904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret") pod "global-pull-secret-syncer-j9lz9" (UID: "bbeed871-241a-4509-ac94-dbe5170904fe") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:44.052359 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.052339    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:44.052465 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.052395    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.052378612 +0000 UTC m=+6.050433595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:44.152530 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.152496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:44.152684 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.152651    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:44.152684 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.152669    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:44.152684 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.152680    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zd9bm for pod openshift-network-diagnostics/network-check-target-qp2mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:44.152829 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.152732    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm podName:ffa11b25-01d3-4d6a-ad70-fcfe34ad324c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.152714016 +0000 UTC m=+6.150768996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zd9bm" (UniqueName: "kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm") pod "network-check-target-qp2mt" (UID: "ffa11b25-01d3-4d6a-ad70-fcfe34ad324c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:44.382119 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.382002    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:42 +0000 UTC" deadline="2028-07-23 09:55:11.526870338 +0000 UTC"
Oct 29 22:13:44.382119 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.382064    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15179h41m27.144810665s"
Oct 29 22:13:44.464356 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.464327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:44.464522 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:44.464457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:44.644697 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.644081    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"e06e61f905fbd53de8ba68af88289e72","Type":"ContainerStarted","Data":"058eab6bb71464574b236721040646b869a81791c49a8f5c4ad4bf3e9ab1c04d"}
Oct 29 22:13:44.682901 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.682844    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" podStartSLOduration=3.682825797 podStartE2EDuration="3.682825797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:43.613077769 +0000 UTC m=+3.611132758" watchObservedRunningTime="2026-10-29 22:13:44.682825797 +0000 UTC m=+4.680880787"
Oct 29 22:13:44.683105 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:44.683068    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" podStartSLOduration=3.683057169 podStartE2EDuration="3.683057169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:44.682293187 +0000 UTC m=+4.680348176" watchObservedRunningTime="2026-10-29 22:13:44.683057169 +0000 UTC m=+4.681112160"
Oct 29 22:13:45.461406 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:45.461264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:45.461406 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:45.461397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:45.461844 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:45.461772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:45.461877 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:45.461861    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:45.648863 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:45.648780    2578 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="56eae3f7161cb32f806bae5ebe7a5ef19aa24a3c3f90811ecc19b8c8049ec54b" exitCode=0
Oct 29 22:13:45.649812 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:45.649764    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerDied","Data":"56eae3f7161cb32f806bae5ebe7a5ef19aa24a3c3f90811ecc19b8c8049ec54b"}
Oct 29 22:13:46.070367 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:46.069603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:46.070367 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:46.069678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:46.070367 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.069808    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:46.070367 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.069864    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:50.069848006 +0000 UTC m=+10.067902991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:46.070367 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.070243    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:46.070367 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.070282    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret podName:bbeed871-241a-4509-ac94-dbe5170904fe nodeName:}" failed. No retries permitted until 2026-10-29 22:13:50.070269244 +0000 UTC m=+10.068324225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret") pod "global-pull-secret-syncer-j9lz9" (UID: "bbeed871-241a-4509-ac94-dbe5170904fe") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:46.171554 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:46.170883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:46.171554 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.171101    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:46.171554 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.171122    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:46.171554 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.171134    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zd9bm for pod openshift-network-diagnostics/network-check-target-qp2mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:46.171554 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.171189    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm podName:ffa11b25-01d3-4d6a-ad70-fcfe34ad324c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:50.171172239 +0000 UTC m=+10.169227212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zd9bm" (UniqueName: "kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm") pod "network-check-target-qp2mt" (UID: "ffa11b25-01d3-4d6a-ad70-fcfe34ad324c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:46.461111 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:46.461080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:46.461264 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:46.461222    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:46.665457 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:46.665382    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x4bzm" event={"ID":"3e49e136-156b-4dfa-a877-51b5c57d0ffa","Type":"ContainerStarted","Data":"073c2cf8e4e9c22dbb25764799184d4c6d4eb481cb50195c864feb6c101dd3cf"}
Oct 29 22:13:47.460874 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:47.460766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:47.461018 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:47.460880    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:47.461356 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:47.461273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:47.461451 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:47.461378    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:47.677086 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:47.676971    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5q62r" event={"ID":"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054","Type":"ContainerStarted","Data":"8f4331e077c3114181c71a59452818a46143dc987d0b99d101e684df5986a102"}
Oct 29 22:13:47.717933 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:47.717304    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x4bzm" podStartSLOduration=4.119261114 podStartE2EDuration="7.717285367s" totalImagesPullingTime="3.598024253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:46.703540173 +0000 UTC m=+6.701595166" watchObservedRunningTime="2026-10-29 22:13:47.717285367 +0000 UTC m=+7.715340359"
Oct 29 22:13:47.980618 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:47.980340    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:47.981257 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:47.981038    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:13:48.017685 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:48.017635    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5q62r" podStartSLOduration=3.455499503 podStartE2EDuration="8.017620164s" totalImagesPullingTime="4.562120661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:47.717826055 +0000 UTC m=+7.715881045" watchObservedRunningTime="2026-10-29 22:13:48.017620164 +0000 UTC m=+8.015675153"
Oct 29 22:13:48.462124 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:48.462094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:48.462261 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:48.462232    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:48.683981 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:48.683515    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x" event={"ID":"0c6470dd-33de-42f4-a144-2fe004a117be","Type":"ContainerStarted","Data":"b164c469266fe028905c6ebf456d05ed990f8804a075829d38cee14a3d49da11"}
Oct 29 22:13:49.461144 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:49.461073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:49.461300 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:49.461209    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:49.461628 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:49.461605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:49.461751 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:49.461728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:49.688448 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:49.687835    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:13:49.786055 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:49.786028    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:13:50.102436 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.102364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:50.102436 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.102407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:50.102665 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.102528    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:50.102665 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.102596    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret podName:bbeed871-241a-4509-ac94-dbe5170904fe nodeName:}" failed. No retries permitted until 2026-10-29 22:13:58.102577352 +0000 UTC m=+18.100632319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret") pod "global-pull-secret-syncer-j9lz9" (UID: "bbeed871-241a-4509-ac94-dbe5170904fe") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:50.102665 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.102529    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:50.102820 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.102675    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:58.102655896 +0000 UTC m=+18.100710905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:50.203272 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.202710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:50.203272 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.202888    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:50.203272 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.202923    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:50.203272 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.202937    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zd9bm for pod openshift-network-diagnostics/network-check-target-qp2mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:50.203272 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.202986    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm podName:ffa11b25-01d3-4d6a-ad70-fcfe34ad324c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:58.202969126 +0000 UTC m=+18.201024098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zd9bm" (UniqueName: "kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm") pod "network-check-target-qp2mt" (UID: "ffa11b25-01d3-4d6a-ad70-fcfe34ad324c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:50.392208 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.392049    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:13:49.78605025Z","UUID":"0d5c6ce3-b433-45ae-aa49-773dc6f0f00d","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:13:50.395201 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.394841    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:13:50.395201 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.394871    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:13:50.462118 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.461719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:50.462118 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:50.461818    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:50.693015 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.692930    2578 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="16472ba9e480d887a154573e6a8f733d676fd268723c2447256cc28c833c6fb0" exitCode=0
Oct 29 22:13:50.693396 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.693013    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerDied","Data":"16472ba9e480d887a154573e6a8f733d676fd268723c2447256cc28c833c6fb0"}
Oct 29 22:13:50.696648 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.696622    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngnk6" event={"ID":"f87a90af-8294-4303-b000-6b4e52289f79","Type":"ContainerStarted","Data":"5feb4840951ecd1b9df83fedbdd82afed1a37d82253a5034a6124ff8c35d72ba"}
Oct 29 22:13:50.699336 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.699308    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7cws" event={"ID":"41bc42bb-f14c-4921-a5f4-5a7ceec653e6","Type":"ContainerStarted","Data":"29b147a0cf42d19191bf5239eaf90b98c09a2c9999041ea9dc0513076d21b1dd"}
Oct 29 22:13:50.701370 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.701349    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x" event={"ID":"0c6470dd-33de-42f4-a144-2fe004a117be","Type":"ContainerStarted","Data":"9ca5d306e56af2d5c2c110258659841dfef582d72d06da054efadd17717e0512"}
Oct 29 22:13:50.773364 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.773316    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-l7cws" podStartSLOduration=3.305715978 podStartE2EDuration="10.773303649s" totalImagesPullingTime="7.467587671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:50.772390727 +0000 UTC m=+10.770445719" watchObservedRunningTime="2026-10-29 22:13:50.773303649 +0000 UTC m=+10.771358629"
Oct 29 22:13:50.811948 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:50.811884    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ngnk6" podStartSLOduration=3.344830478 podStartE2EDuration="10.811871494s" totalImagesPullingTime="7.467041016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:50.811193374 +0000 UTC m=+10.809248366" watchObservedRunningTime="2026-10-29 22:13:50.811871494 +0000 UTC m=+10.809926483"
Oct 29 22:13:51.461611 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:51.461587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:51.461711 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:51.461634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:51.461711 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:51.461700    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:51.461844 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:51.461813    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:51.706241 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:51.706155    2578 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="173eb099487ce5887ee4b3467561d6bf9ad6dc109b364d898398cae44a01c93e" exitCode=0
Oct 29 22:13:51.706635 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:51.706255    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerDied","Data":"173eb099487ce5887ee4b3467561d6bf9ad6dc109b364d898398cae44a01c93e"}
Oct 29 22:13:51.709103 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:51.709071    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x" event={"ID":"0c6470dd-33de-42f4-a144-2fe004a117be","Type":"ContainerStarted","Data":"ba1902f0677b513b18cb343b05b6642d673f667438459a19d92c2da0dcfe6979"}
Oct 29 22:13:51.799187 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:51.799129    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x" podStartSLOduration=4.083536183 podStartE2EDuration="11.799110444s" totalImagesPullingTime="7.715574261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:51.798153366 +0000 UTC m=+11.796208359" watchObservedRunningTime="2026-10-29 22:13:51.799110444 +0000 UTC m=+11.797165437"
Oct 29 22:13:52.461047 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:52.460985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:52.461243 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:52.461099    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:52.712551 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:52.712492    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7nhrb" event={"ID":"26757508-3500-4597-b243-ae17376bd037","Type":"ContainerStarted","Data":"5771d46e72352f9a0a8398eeeca3461c9e8d86578d2a2a72129420b9c9c98b87"}
Oct 29 22:13:52.714662 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:52.714635    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wx7df" event={"ID":"95a82d11-1674-4b89-bdeb-6fbf6caa5304","Type":"ContainerStarted","Data":"4838bac3831c3fdf50bc582000cf920ae5963799c97ea8a172cc2fcd6a5ac080"}
Oct 29 22:13:52.750550 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:52.750510    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7nhrb" podStartSLOduration=3.283860674 podStartE2EDuration="12.750497121s" totalImagesPullingTime="9.466636447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:52.749847427 +0000 UTC m=+12.747902417" watchObservedRunningTime="2026-10-29 22:13:52.750497121 +0000 UTC m=+12.748552109"
Oct 29 22:13:53.461757 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:53.461716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:53.461757 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:53.461750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:53.461988 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:53.461856    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:53.462060 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:53.462023    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:53.719506 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:53.719416    2578 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="3987d4d5ffe75938181051b92233774d9bc82f45213d7873d022c772c4953739" exitCode=0
Oct 29 22:13:53.720053 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:53.719515    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerDied","Data":"3987d4d5ffe75938181051b92233774d9bc82f45213d7873d022c772c4953739"}
Oct 29 22:13:53.786256 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:53.786210    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wx7df" podStartSLOduration=6.006715907 podStartE2EDuration="13.786195583s" totalImagesPullingTime="7.779479676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:52.787077404 +0000 UTC m=+12.785132394" watchObservedRunningTime="2026-10-29 22:13:53.786195583 +0000 UTC m=+13.784250566"
Oct 29 22:13:54.461175 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:54.461145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:54.461329 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:54.461260    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:55.461746 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:55.461561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:55.462234 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:55.461613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:55.462234 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:55.461846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:55.462234 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:55.461900    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:55.727305 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:55.727211    2578 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="ec961f153bb20754761e6ef61c91d5d398428e096ed997751036daed8fabca45" exitCode=0
Oct 29 22:13:55.727305 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:55.727247    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerDied","Data":"ec961f153bb20754761e6ef61c91d5d398428e096ed997751036daed8fabca45"}
Oct 29 22:13:56.461616 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:56.461582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:56.461748 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:56.461713    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:56.731397 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:56.731326    2578 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="5fae997e117515b904b16fa38fe02e7e1a56aed517611a6043353f24bb78c1df" exitCode=0
Oct 29 22:13:56.731397 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:56.731369    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerDied","Data":"5fae997e117515b904b16fa38fe02e7e1a56aed517611a6043353f24bb78c1df"}
Oct 29 22:13:57.461505 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.461483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:57.461619 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:57.461600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:57.461662 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.461631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:57.461697 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:57.461688    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:57.736452 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.736366    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"91ed24981bfe0322f89cf4c8d7e5867ba3473ab188d0b4c0b42957e94356f7f1"}
Oct 29 22:13:57.736452 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.736436    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"1c6f944695a8d3b59165535245928a8fd2173e522fa0d2f9768752305be90508"}
Oct 29 22:13:57.736885 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.736475    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"0f62a5c7d6b0708a4b64c4bc647ec2a0a1cfb2e02df619fe14ab2cc30ff43dcc"}
Oct 29 22:13:57.736885 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.736505    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"665923676b17b94878343f54743f3c34a4eed5d1a1c944f6b06f979ce3eb224d"}
Oct 29 22:13:57.736885 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.736536    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"d9dd0db34bc0ca81def0f00d1a36cada0e4c5d6939b35cc9238962842379e276"}
Oct 29 22:13:57.736885 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.736570    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"8a5f1a335c1efb6c308f58e56507636a76bd173fcf498649360dc6824073a11b"}
Oct 29 22:13:57.739710 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.739690    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" event={"ID":"30bf6468-e2cc-4f55-a719-c33ab521bf43","Type":"ContainerStarted","Data":"3ca56344046879f1813d79483a253e02e110e9bdc3bbf16016ea2316381fc91d"}
Oct 29 22:13:57.780666 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:57.780604    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hnw4v" podStartSLOduration=7.470517441 podStartE2EDuration="17.780592521s" totalImagesPullingTime="10.31007508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:57.780008548 +0000 UTC m=+17.778063537" watchObservedRunningTime="2026-10-29 22:13:57.780592521 +0000 UTC m=+17.778647510"
Oct 29 22:13:58.164778 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:58.164700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:58.164778 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:58.164747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:58.164981 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.164829    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:58.164981 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.164834    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:58.164981 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.164876    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.164862737 +0000 UTC m=+34.162917704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:58.164981 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.164888    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret podName:bbeed871-241a-4509-ac94-dbe5170904fe nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.164882658 +0000 UTC m=+34.162937626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret") pod "global-pull-secret-syncer-j9lz9" (UID: "bbeed871-241a-4509-ac94-dbe5170904fe") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:58.264948 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:58.264901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:58.265061 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.265048    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:58.265097 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.265064    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:58.265097 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.265073    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zd9bm for pod openshift-network-diagnostics/network-check-target-qp2mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:58.265154 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.265115    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm podName:ffa11b25-01d3-4d6a-ad70-fcfe34ad324c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.265102925 +0000 UTC m=+34.263157893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zd9bm" (UniqueName: "kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm") pod "network-check-target-qp2mt" (UID: "ffa11b25-01d3-4d6a-ad70-fcfe34ad324c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:58.460858 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:58.460793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:13:58.460994 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:58.460885    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:13:59.461566 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:59.461539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:13:59.462097 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:59.461539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:13:59.462097 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:59.461657    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:13:59.462097 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:13:59.461694    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:13:59.750095 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:13:59.750018    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"15f159785abcc1280484c4cf2d6501a680ca0d9758cb31ee9ecbcb194c833082"}
Oct 29 22:14:00.461974 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:00.461946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:00.462330 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:00.462028    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:14:01.461125 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.461098    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:01.461125 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.461127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:01.461309 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:01.461198    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:14:01.461309 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:01.461252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:14:01.758218 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.758183    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" event={"ID":"7520d944-da53-4455-868f-3880e5535345","Type":"ContainerStarted","Data":"4a66ffc8261e79b835d541482d4fd077ef293f4c54a71ab3e9387c40640ac0cc"}
Oct 29 22:14:01.758862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.758511    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:01.758862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.758640    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:01.758862 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.758693    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:01.773763 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.773739    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:01.774167 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.774149    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:01.839155 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:01.839101    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74" podStartSLOduration=7.367013843 podStartE2EDuration="21.83908796s" totalImagesPullingTime="14.472074117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:01.83841629 +0000 UTC m=+21.836471306" watchObservedRunningTime="2026-10-29 22:14:01.83908796 +0000 UTC m=+21.837142949"
Oct 29 22:14:02.461240 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:02.461211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:02.461381 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:02.461313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:14:03.461469 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.461237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:03.461822 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.461252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:03.461822 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:03.461600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:14:03.461822 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:03.461698    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:14:03.719619 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.719493    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qp2mt"]
Oct 29 22:14:03.720115 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.720091    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j9lz9"]
Oct 29 22:14:03.720234 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.720177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:03.720578 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:03.720268    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:14:03.730645 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.730622    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g7bbj"]
Oct 29 22:14:03.761501 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.761480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:03.761594 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:03.761487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:03.761594 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:03.761567    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:14:03.761964 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:03.761942    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:14:05.460724 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:05.460696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:05.461138 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:05.460789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:05.461138 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:05.460793    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:14:05.461138 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:05.460877    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:14:05.461138 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:05.460900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:05.461138 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:05.460973    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:14:07.460764 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:07.460735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:07.461132 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:07.460844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:07.461132 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:07.460858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:07.461132 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:07.460863    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:14:07.461132 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:07.460941    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:14:07.461132 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:07.460999    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:14:09.461701 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.461623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:09.461701 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.461655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:09.461701 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.461628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:09.462200 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:09.461748    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9lz9" podUID="bbeed871-241a-4509-ac94-dbe5170904fe"
Oct 29 22:14:09.462200 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:09.461891    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7bbj" podUID="e36a49f7-9cf1-4bd1-aed4-c66384880090"
Oct 29 22:14:09.462200 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:09.461986    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qp2mt" podUID="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c"
Oct 29 22:14:09.804343 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.804313    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeReady"
Oct 29 22:14:09.804484 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.804429    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:14:09.857798 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.857763    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nftxr"]
Oct 29 22:14:09.922167 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.922138    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2lqth"]
Oct 29 22:14:09.922326 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.922251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:09.935054 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.935035    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nftxr"]
Oct 29 22:14:09.935163 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.935065    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:09.935163 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.935102    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:09.935163 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.935116    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2lqth"]
Oct 29 22:14:09.935163 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.935141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:09.937082 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.937065    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:14:09.937214 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.937082    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:14:09.937214 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.937169    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xsn6t\""
Oct 29 22:14:09.937214 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.937191    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:14:09.948796 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.948780    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:14:09.949019 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.949003    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:14:09.949088 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:09.949076    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nqlz\""
Oct 29 22:14:10.052649 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.052565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb945a9-e793-4e53-bb13-38b3eb5022e4-config-volume\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.052649 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.052609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.052649 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.052631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbrh\" (UniqueName: \"kubernetes.io/projected/51fd9e7b-409f-4896-b369-ca78469fc468-kube-api-access-vhbrh\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.052857 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.052764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb945a9-e793-4e53-bb13-38b3eb5022e4-tmp-dir\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.052857 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.052796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51fd9e7b-409f-4896-b369-ca78469fc468-cert\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.052958 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.052887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zsx\" (UniqueName: \"kubernetes.io/projected/7bb945a9-e793-4e53-bb13-38b3eb5022e4-kube-api-access-88zsx\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.153424 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51fd9e7b-409f-4896-b369-ca78469fc468-cert\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.153511 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zsx\" (UniqueName: \"kubernetes.io/projected/7bb945a9-e793-4e53-bb13-38b3eb5022e4-kube-api-access-88zsx\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.153511 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb945a9-e793-4e53-bb13-38b3eb5022e4-config-volume\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.153588 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.153588 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbrh\" (UniqueName: \"kubernetes.io/projected/51fd9e7b-409f-4896-b369-ca78469fc468-kube-api-access-vhbrh\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.153661 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb945a9-e793-4e53-bb13-38b3eb5022e4-tmp-dir\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.153661 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:10.153645    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:10.153735 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:10.153719    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:10.653700876 +0000 UTC m=+30.651755869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:10.153952 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.153934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb945a9-e793-4e53-bb13-38b3eb5022e4-tmp-dir\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.154191 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.154173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb945a9-e793-4e53-bb13-38b3eb5022e4-config-volume\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.157172 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.157153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51fd9e7b-409f-4896-b369-ca78469fc468-cert\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.172289 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.172267    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zsx\" (UniqueName: \"kubernetes.io/projected/7bb945a9-e793-4e53-bb13-38b3eb5022e4-kube-api-access-88zsx\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.172568 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.172551    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbrh\" (UniqueName: \"kubernetes.io/projected/51fd9e7b-409f-4896-b369-ca78469fc468-kube-api-access-vhbrh\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.243878 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.243855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:10.602562 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.602301    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nftxr"]
Oct 29 22:14:10.604995 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:14:10.604967    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fd9e7b_409f_4896_b369_ca78469fc468.slice/crio-085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d WatchSource:0}: Error finding container 085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d: Status 404 returned error can't find the container with id 085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d
Oct 29 22:14:10.657151 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.657123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:10.657279 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:10.657263    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:10.657328 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:10.657314    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.657300122 +0000 UTC m=+31.655355094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:10.777643 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:10.777618    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nftxr" event={"ID":"51fd9e7b-409f-4896-b369-ca78469fc468","Type":"ContainerStarted","Data":"085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d"}
Oct 29 22:14:11.461492 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.461457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:11.461492 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.461490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:11.461691 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.461631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:11.476499 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.476462    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:14:11.488292 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.488273    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7vbxc\""
Oct 29 22:14:11.488398 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.488281    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:14:11.488398 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.488319    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:14:11.488398 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.488320    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:14:11.488398 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.488393    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v7npq\""
Oct 29 22:14:11.488522 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.488437    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-977wj\""
Oct 29 22:14:11.664309 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:11.664276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:11.664727 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:11.664382    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:11.664727 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:11.664439    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:13.664422223 +0000 UTC m=+33.662477199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:13.677713 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:13.677674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:13.678139 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:13.677820    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:13.678139 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:13.677886    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:17.677870666 +0000 UTC m=+37.675925647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:13.786619 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:13.786591    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nftxr" event={"ID":"51fd9e7b-409f-4896-b369-ca78469fc468","Type":"ContainerStarted","Data":"a602a5b1a31b3a6da19631de9f572a2a72b653369a243a6c151e1ff8867f4498"}
Oct 29 22:14:13.823291 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:13.823249    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nftxr" podStartSLOduration=2.298948879 podStartE2EDuration="4.823237357s" totalImagesPullingTime="2.524288478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.822158776 +0000 UTC m=+33.820213777" watchObservedRunningTime="2026-10-29 22:14:13.823237357 +0000 UTC m=+33.821292343"
Oct 29 22:14:14.181658 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.181580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:14.181658 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.181626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:14.181838 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.181723    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:14.181838 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.181796    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.181781076 +0000 UTC m=+66.179836044 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : secret "metrics-daemon-secret" not found
Oct 29 22:14:14.183809 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.183784    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:14.188227 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.188205    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qvpbn"]
Oct 29 22:14:14.221652 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.221633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:14.237136 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.237116    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qvpbn"]
Oct 29 22:14:14.237223 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.237212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.282711 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.282679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:14.285003 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.284978    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:14.324326 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.324302    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:14:14.324326 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.324317    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:14:14.324519 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.324505    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:14:14.324707 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.324694    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:14:14.324903 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.324890    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpwxr\""
Oct 29 22:14:14.383174 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.383150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6dd904-33a2-4497-9424-654062e29812-data-volume\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.383261 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.383203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6dd904-33a2-4497-9424-654062e29812-crio-socket\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.383305 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.383287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.383345 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.383317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.383345 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.383335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdhz\" (UniqueName: \"kubernetes.io/projected/1b6dd904-33a2-4497-9424-654062e29812-kube-api-access-2vdhz\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484082 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484018    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"]
Oct 29 22:14:14.484206 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6dd904-33a2-4497-9424-654062e29812-data-volume\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484254 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6dd904-33a2-4497-9424-654062e29812-crio-socket\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484287 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484287 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484356 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdhz\" (UniqueName: \"kubernetes.io/projected/1b6dd904-33a2-4497-9424-654062e29812-kube-api-access-2vdhz\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484419 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.484402    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.484487 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.484473    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.984455847 +0000 UTC m=+34.982510834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.484548 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484517    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6dd904-33a2-4497-9424-654062e29812-crio-socket\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484603 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.484540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6dd904-33a2-4497-9424-654062e29812-data-volume\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.484639 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.484616    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.484670 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.484664    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.984653658 +0000 UTC m=+34.982708650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.500843 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.500822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:14.506512 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.506492    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"]
Oct 29 22:14:14.509239 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.509219    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdhz\" (UniqueName: \"kubernetes.io/projected/1b6dd904-33a2-4497-9424-654062e29812-kube-api-access-2vdhz\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.517234 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.517211    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:14.517325 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.517263    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 22:14:14.517325 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.517295    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kj4hx\""
Oct 29 22:14:14.537861 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.537837    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j9lz9"]
Oct 29 22:14:14.541121 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.541100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:14.543765 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:14:14.543738    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeed871_241a_4509_ac94_dbe5170904fe.slice/crio-0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc WatchSource:0}: Error finding container 0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc: Status 404 returned error can't find the container with id 0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc
Oct 29 22:14:14.584796 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.584774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtsz\" (UniqueName: \"kubernetes.io/projected/8a5332a6-581b-40bd-ba3c-284bcece1df1-kube-api-access-dhtsz\") pod \"migrator-66df5d496b-4jbpj\" (UID: \"8a5332a6-581b-40bd-ba3c-284bcece1df1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:14.685707 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.685676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtsz\" (UniqueName: \"kubernetes.io/projected/8a5332a6-581b-40bd-ba3c-284bcece1df1-kube-api-access-dhtsz\") pod \"migrator-66df5d496b-4jbpj\" (UID: \"8a5332a6-581b-40bd-ba3c-284bcece1df1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:14.703952 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.703924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtsz\" (UniqueName: \"kubernetes.io/projected/8a5332a6-581b-40bd-ba3c-284bcece1df1-kube-api-access-dhtsz\") pod \"migrator-66df5d496b-4jbpj\" (UID: \"8a5332a6-581b-40bd-ba3c-284bcece1df1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:14.790002 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.789936    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j9lz9" event={"ID":"bbeed871-241a-4509-ac94-dbe5170904fe","Type":"ContainerStarted","Data":"0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc"}
Oct 29 22:14:14.831697 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.831672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:14.850093 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.850071    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qp2mt"]
Oct 29 22:14:14.852892 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:14:14.852869    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa11b25_01d3_4d6a_ad70_fcfe34ad324c.slice/crio-5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c WatchSource:0}: Error finding container 5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c: Status 404 returned error can't find the container with id 5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c
Oct 29 22:14:14.987634 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.987598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.987761 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:14.987647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:14.987801 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.987769    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.987801 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.987782    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.987862 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.987856    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.987820295 +0000 UTC m=+35.985875280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.987902 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:14.987879    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.987868558 +0000 UTC m=+35.985923530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.140230 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:15.140209    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"]
Oct 29 22:14:15.142400 ip-10-0-143-15 kubenswrapper[2578]: W1029 22:14:15.142376    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5332a6_581b_40bd_ba3c_284bcece1df1.slice/crio-811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b WatchSource:0}: Error finding container 811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b: Status 404 returned error can't find the container with id 811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b
Oct 29 22:14:15.795586 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:15.795545    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qp2mt" event={"ID":"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c","Type":"ContainerStarted","Data":"5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c"}
Oct 29 22:14:15.798369 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:15.798344    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj" event={"ID":"8a5332a6-581b-40bd-ba3c-284bcece1df1","Type":"ContainerStarted","Data":"811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b"}
Oct 29 22:14:15.994592 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:15.994554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:15.994592 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:15.994605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:15.994834 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:15.994695    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.994834 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:15.994754    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:17.994737366 +0000 UTC m=+37.992792341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.995105 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:15.995080    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:15.995215 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:15.995144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:17.995128031 +0000 UTC m=+37.993183001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:17.705361 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:17.705326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:17.705808 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:17.705479    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:17.705808 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:17.705536    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:25.705521599 +0000 UTC m=+45.703576578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:17.805995 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:17.805947    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj" event={"ID":"8a5332a6-581b-40bd-ba3c-284bcece1df1","Type":"ContainerStarted","Data":"2271398ec9a2635bf8e269c4b95b86a7ad21fafe724d47770b92df0a13d0f39e"}
Oct 29 22:14:17.843400 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:17.843334    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj" podStartSLOduration=2.248181084 podStartE2EDuration="3.843316374s" totalImagesPullingTime="1.59513529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:17.842853365 +0000 UTC m=+37.840908373" watchObservedRunningTime="2026-10-29 22:14:17.843316374 +0000 UTC m=+37.841371365"
Oct 29 22:14:18.007868 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:18.007788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:18.007868 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:18.007839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:18.008064 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:18.007959    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:18.008064 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:18.007973    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:18.008064 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:18.008032    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.008016228 +0000 UTC m=+42.006071211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:18.008064 ip-10-0-143-15 kubenswrapper[2578]: E1029 22:14:18.008053    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls podName:1b6dd904-33a2-4497-9424-654062e29812 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.008044005 +0000 UTC m=+42.006098997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvpbn" (UID: "1b6dd904-33a2-4497-9424-654062e29812") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:18.810472 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:18.810394    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qp2mt" event={"ID":"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c","Type":"ContainerStarted","Data":"d390e04bec1b02d42a6c448d313ac2a09e7ebe21902b9c2ee4e5f30221c4356c"}
Oct 29 22:14:18.849384 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:18.849324    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qp2mt" podStartSLOduration=35.334213736 podStartE2EDuration="38.849305534s" totalImagesPullingTime="3.515091798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:18.848234233 +0000 UTC m=+38.846289224" watchObservedRunningTime="2026-10-29 22:14:18.849305534 +0000 UTC m=+38.847360526"
Oct 29 22:14:19.814687 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:19.814654    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j9lz9" event={"ID":"bbeed871-241a-4509-ac94-dbe5170904fe","Type":"ContainerStarted","Data":"c93c12272946e641536937ca4110a5d31009c1ff15a89b5645479a7aa9b6576f"}
Oct 29 22:14:19.815141 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:19.814793    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:19.853619 ip-10-0-143-15 kubenswrapper[2578]: I1029 22:14:19.853573    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j9lz9" podStartSLOduration=34.728784329 podStartE2EDuration="39.85355767s" totalImagesPullingTime="5.124773341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:19.853102787 +0000 UTC m=+39.851157776" watchObservedRunningTime="2026-10-29 22:14:19.85355767 +0000 UTC m=+39.851612659"
Oct 29 22:14:19.969191 ip-10-0-143-15 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:14:19.982288 ip-10-0-143-15 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:14:19.982452 ip-10-0-143-15 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:14:19.982499 ip-10-0-143-15 systemd[1]: kubelet.service: Consumed 2.137s CPU time.
Oct 29 22:14:19.997106 ip-10-0-143-15 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:14:20.086343 ip-10-0-143-15 kubenswrapper[4451]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:20.086343 ip-10-0-143-15 kubenswrapper[4451]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:14:20.086343 ip-10-0-143-15 kubenswrapper[4451]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:20.086343 ip-10-0-143-15 kubenswrapper[4451]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:20.088460 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088379    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.088460 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088459    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.088460 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088464    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088467    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088471    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088474    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088477    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088480    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088482    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088485    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088487    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088490    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088492    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088501    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088504    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088507    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088509    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088512    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088515    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088517    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088520    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088522    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.088561 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088525    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088527    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088530    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088533    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088536    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088538    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088541    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088543    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088546    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088549    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088551    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088554    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088557    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088560    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088563    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088565    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088568    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088571    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088573    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.089102 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088578    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088582    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088586    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088589    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088592    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088596    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088598    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088601    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088604    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088606    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088609    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088612    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088614    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088617    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088619    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088622    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088625    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088627    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088629    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.089568 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088633    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088636    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088639    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088641    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088644    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088646    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088649    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088652    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088654    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088657    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088660    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088662    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088664    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088667    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088669    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088672    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088675    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088677    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088682    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088686    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.090054 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088688    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088692    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088694    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088697    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088700    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088702    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088705    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088707    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088713    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088716    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088718    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088721    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088723    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088726    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088730    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088732    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088735    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088737    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088739    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.090536 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088743    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088746    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088748    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088751    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088753    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088756    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.088758    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089536    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089544    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089547    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089551    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089553    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089556    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089559    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089562    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089565    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089567    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089570    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089573    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.091023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089575    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089578    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089581    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089584    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089587    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089589    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089592    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089595    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089597    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089600    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089603    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089605    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089608    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089611    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089613    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089616    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089619    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089622    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089624    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.091505 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089627    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089630    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089632    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089635    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089638    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089640    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089643    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089645    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089648    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089651    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089654    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089657    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089659    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089662    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089664    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089667    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089670    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089672    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089675    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089677    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.091972 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089697    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089701    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089705    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089708    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089711    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089713    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089716    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089719    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089721    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089726    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089731    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089734    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089737    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089739    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089742    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089745    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089747    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089750    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089753    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.092455 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089755    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089758    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089761    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089763    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089766    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089769    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089773    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089776    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089779    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089783    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089785    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089788    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089791    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089794    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089796    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089799    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089801    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089804    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089806    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089810    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.092940 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089812    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089815    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089817    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089820    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089822    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089825    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089827    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089830    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089832    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089835    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089837    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089840    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089842    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089845    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089847    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.089850    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090406    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090413    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090417    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090420    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.093424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090423    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090426    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090429    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090432    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090435    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090437    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090440    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090442    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090445    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090448    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090451    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090453    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090456    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090459    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090461    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090464    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090466    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090469    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090471    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.093966 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090474    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090479    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090481    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090484    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090486    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090489    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090491    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090493    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090504    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090507    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090509    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090512    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090514    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090517    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090519    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090522    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090524    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090527    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090529    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090532    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.094432 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090534    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090537    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090540    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090542    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090545    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090548    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090551    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090554    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090556    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090558    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090561    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090563    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090566    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090570    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090573    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090576    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090578    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090581    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090583    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090586    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.094936 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090588    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090596    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090599    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090602    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090604    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090607    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090609    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090612    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090615    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090617    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090620    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090622    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090625    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090627    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090630    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090633    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090635    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090638    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090640    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.095421 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090643    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090645    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090648    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090652    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090655    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090658    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090663    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090667    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090669    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090672    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090674    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090677    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090680    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090682    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090684    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090694    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090696    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090699    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090701    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.095882 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090704    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090706    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090709    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090711    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.090715    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090806    4451 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090813    4451 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090825    4451 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090830    4451 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090835    4451 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090838    4451 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090842    4451 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090846    4451 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090849    4451 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090852    4451 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090855    4451 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090858    4451 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090861    4451 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090864    4451 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090868    4451 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090872    4451 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090875    4451 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090878    4451 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:14:20.096359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090884    4451 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090887    4451 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090890    4451 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090893    4451 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090897    4451 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090900    4451 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090921    4451 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090925    4451 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090928    4451 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090931    4451 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090934    4451 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090937    4451 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090940    4451 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090943    4451 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090947    4451 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090950    4451 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090952    4451 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090957    4451 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090960    4451 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090963    4451 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090970    4451 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090974    4451 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090977    4451 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090985    4451 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:14:20.097078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090989    4451 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090993    4451 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090996    4451 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.090999    4451 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091004    4451 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091007    4451 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091010    4451 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091013    4451 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091016    4451 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091019    4451 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091022    4451 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091025    4451 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091029    4451 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091032    4451 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091036    4451 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091044    4451 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091048    4451 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091051    4451 flags.go:64] FLAG: --help="false"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091054    4451 flags.go:64] FLAG: --hostname-override="ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091057    4451 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091060    4451 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091063    4451 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091066    4451 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:14:20.097900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091069    4451 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091072    4451 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091077    4451 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091080    4451 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091082    4451 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091085    4451 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091089    4451 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091092    4451 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091095    4451 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091098    4451 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091101    4451 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091104    4451 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091106    4451 flags.go:64] FLAG: --lock-file=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091111    4451 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091114    4451 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091117    4451 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091122    4451 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091125    4451 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091128    4451 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091131    4451 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091133    4451 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091137    4451 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091140    4451 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091142    4451 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091147    4451 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:14:20.098483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091155    4451 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091160    4451 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091163    4451 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091165    4451 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091168    4451 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091171    4451 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091175    4451 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091178    4451 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091181    4451 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091189    4451 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091192    4451 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091195    4451 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091198    4451 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091201    4451 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091204    4451 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091207    4451 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091210    4451 flags.go:64] FLAG: --port="10250"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091213    4451 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091216    4451 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f2e8fdfa2b152b75"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091219    4451 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091224    4451 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091227    4451 flags.go:64] FLAG: --register-node="true"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091230    4451 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091234    4451 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091236    4451 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091239    4451 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:14:20.099147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091242    4451 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091246    4451 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091249    4451 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091252    4451 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091255    4451 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091258    4451 flags.go:64] FLAG: --runonce="false"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091261    4451 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091270    4451 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091273    4451 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091276    4451 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091279    4451 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091282    4451 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091285    4451 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091288    4451 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091291    4451 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091295    4451 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091298    4451 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091301    4451 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091304    4451 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091307    4451 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091310    4451 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091315    4451 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091318    4451 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091321    4451 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091334    4451 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:14:20.099857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091337    4451 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091341    4451 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091344    4451 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091347    4451 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091350    4451 flags.go:64] FLAG: --v="2"
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091355    4451 flags.go:64] FLAG: --version="false"
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091359    4451 flags.go:64] FLAG: --vmodule=""
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091363    4451 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091367    4451 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091557    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091560    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091563    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091569    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091572    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091575    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091584    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091587    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091591    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091595    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091598    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091601    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.100507 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091604    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091608    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091611    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091613    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091616    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091618    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091621    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091624    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091626    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091629    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091632    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091634    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091638    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091641    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091645    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091648    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091651    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091654    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091657    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091660    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.101070 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091663    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091666    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091669    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091671    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091674    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091676    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091680    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091690    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091695    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091698    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091700    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091703    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091706    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091710    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091712    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091715    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091717    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091720    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091722    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.101601 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091725    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091727    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091730    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091733    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091735    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091740    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091743    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091745    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091748    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091750    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091753    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091756    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091758    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091761    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091763    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091766    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091768    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091771    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091773    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091776    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.102197 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091779    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091787    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091790    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091792    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091795    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091797    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091801    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091804    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091806    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091809    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091811    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091814    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091816    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091819    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091822    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091824    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091826    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091831    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091834    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.102737 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091837    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091839    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091842    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091844    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091847    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091849    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091852    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091854    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091857    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091859    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091862    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091865    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091867    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091870    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091872    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091880    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.103266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.091924    4451 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091974    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091978    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091982    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091986    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091989    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091992    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091994    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091997    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.091999    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092002    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092005    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092008    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092010    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092014    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092017    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092019    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092022    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092025    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092027    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.103655 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092030    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092033    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092035    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092037    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092040    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092043    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092045    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092048    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092051    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092053    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092055    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092058    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092067    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092070    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092072    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092075    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092079    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092082    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092084    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.104348 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092087    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092090    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092093    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092095    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092098    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092100    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092103    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092107    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092110    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092112    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092115    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092117    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092120    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092122    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092125    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092127    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092130    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092132    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092135    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092138    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.104903 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092140    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092143    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092146    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092148    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092151    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092154    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092162    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092165    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092167    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092176    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092181    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092185    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092189    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092192    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092195    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092198    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092200    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092203    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092205    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.105420 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092209    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092212    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092214    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092217    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092220    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092223    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092226    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092228    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092231    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092233    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092237    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092240    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092243    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092246    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092248    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092251    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092253    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092256    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092258    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092261    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.105872 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092270    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092273    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092277    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092280    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092283    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092285    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092288    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092291    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.092293    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.096129    4451 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.096144    4451 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.096162    4451 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.096167    4451 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096247    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096253    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096256    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096259    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096262    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096264    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096267    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.106378 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096270    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096272    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096275    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096278    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096280    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096283    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096285    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096288    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096290    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096293    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096296    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096298    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096301    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096304    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096307    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096310    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096312    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096316    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096319    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096321    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096324    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096326    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096333    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096336    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096339    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096342    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096344    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096349    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096353    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096356    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096359    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096362    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096364    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096368    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096371    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096373    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096376    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096379    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096382    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.107369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096385    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096388    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096391    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096393    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096396    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096398    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096401    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096404    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096407    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096410    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096412    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096415    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096417    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096420    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096423    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096427    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096431    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096435    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096437    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096440    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.107817 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096443    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096445    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096448    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096451    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096454    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096456    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096459    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096461    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096464    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096466    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096469    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096472    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096474    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096477    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096479    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096482    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096485    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096487    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096490    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.108364 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096492    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096495    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096498    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096500    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096503    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096505    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096508    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096510    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096513    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096515    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096518    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096520    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096523    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096525    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096528    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096530    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096533    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096535    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096538    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096540    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.108826 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.096543    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.096570    4451 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097081    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097097    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097103    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097109    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097113    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097118    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097122    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097132    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097137    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097141    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097146    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097150    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097155    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097159    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.109305 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097163    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097168    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097172    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097176    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097180    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097184    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097193    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097207    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097212    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097216    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097220    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097224    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097228    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097232    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097237    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097241    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097245    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097250    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097254    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097263    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.109697 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097267    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097271    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097278    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097284    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097289    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097294    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097298    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097302    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097306    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097311    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097315    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097324    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097327    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097332    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097336    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097343    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097349    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097354    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097359    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.110192 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097363    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097368    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097373    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097377    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097381    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097390    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097394    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097398    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097402    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097407    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097411    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097415    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097420    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097424    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097428    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097432    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097436    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097446    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097449    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.110639 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097453    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097458    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097462    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097466    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097470    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097475    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097479    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097483    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097488    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097492    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097496    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097506    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097510    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097514    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097518    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097523    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097527    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097532    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097536    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.111120 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097540    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097544    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097549    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097553    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097563    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097567    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097571    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097576    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097580    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097584    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097588    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097592    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097596    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097600    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097605    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.097661    4451 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:20.111576 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097923    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097932    4451 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097937    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097940    4451 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097943    4451 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097945    4451 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097950    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097953    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097956    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097958    4451 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097961    4451 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097963    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097966    4451 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097970    4451 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097973    4451 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097975    4451 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097978    4451 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097981    4451 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097983    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097986    4451 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:20.111981 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097988    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097991    4451 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097994    4451 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097996    4451 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.097999    4451 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098001    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098004    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098006    4451 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098009    4451 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098012    4451 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098014    4451 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098017    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098019    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098022    4451 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098025    4451 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098027    4451 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098030    4451 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098033    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098037    4451 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:20.112470 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098040    4451 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098042    4451 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098045    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098047    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098050    4451 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098055    4451 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098058    4451 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098061    4451 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098063    4451 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098066    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098068    4451 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098071    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098073    4451 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098075    4451 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098078    4451 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098080    4451 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098083    4451 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098085    4451 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098088    4451 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098090    4451 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:20.113110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098093    4451 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098096    4451 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098098    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098101    4451 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098103    4451 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098106    4451 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098112    4451 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098115    4451 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098117    4451 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098120    4451 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098123    4451 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098126    4451 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098129    4451 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098132    4451 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098134    4451 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098137    4451 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098140    4451 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098143    4451 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098146    4451 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098150    4451 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:20.113598 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098155    4451 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098158    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098160    4451 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098163    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098165    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098168    4451 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098171    4451 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098173    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098176    4451 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098178    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098180    4451 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098183    4451 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098187    4451 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098190    4451 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098193    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098196    4451 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098198    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098201    4451 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098205    4451 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:20.114355 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098208    4451 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098210    4451 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098214    4451 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098217    4451 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098221    4451 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098223    4451 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098226    4451 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:20.098228    4451 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.098291    4451 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099239    4451 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099283    4451 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099450    4451 server.go:1014] "Starting client certificate rotation"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099589    4451 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099639    4451 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:08:40 +0000 UTC" deadline="2028-04-12 09:44:42.351631948 +0000 UTC"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099677    4451 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12731h30m22.251958091s"
Oct 29 22:14:20.115169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.099858    4451 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:20.115764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.100871    4451 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:20.115764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.102278    4451 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:14:20.115764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.103833    4451 log.go:25] "Validated CRI v1 image API"
Oct 29 22:14:20.115764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.104609    4451 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:14:20.115764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.106059    4451 fs.go:135] Filesystem UUIDs: map[5847d7cd-75dc-450c-b0fa-a8ba71e0f63a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cbd286bb-b1a4-4625-8f14-6f34f2f81848:/dev/nvme0n1p3]
Oct 29 22:14:20.116010 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.106079    4451 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054/volumes/kubernetes.io~projected/kube-api-access-5hdhq:{mountpoint:/var/lib/kubelet/pods/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054/volumes/kubernetes.io~projected/kube-api-access-5hdhq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c6470dd-33de-42f4-a144-2fe004a117be/volumes/kubernetes.io~projected/kube-api-access-88r8z:{mountpoint:/var/lib/kubelet/pods/0c6470dd-33de-42f4-a144-2fe004a117be/volumes/kubernetes.io~projected/kube-api-access-88r8z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b6dd904-33a2-4497-9424-654062e29812/volumes/kubernetes.io~projected/kube-api-access-2vdhz:{mountpoint:/var/lib/kubelet/pods/1b6dd904-33a2-4497-9424-654062e29812/volumes/kubernetes.io~projected/kube-api-access-2vdhz major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~projected/kube-api-access-bt7bd:{mountpoint:/var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~projected/kube-api-access-bt7bd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30bf6468-e2cc-4f55-a719-c33ab521bf43/volumes/kubernetes.io~projected/kube-api-access-ksbtf:{mountpoint:/var/lib/kubelet/pods/30bf6468-e2cc-4f55-a719-c33ab521bf43/volumes/kubernetes.io~projected/kube-api-access-ksbtf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e49e136-156b-4dfa-a877-51b5c57d0ffa/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3e49e136-156b-4dfa-a877-51b5c57d0ffa/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41bc42bb-f14c-4921-a5f4-5a7ceec653e6/volumes/kubernetes.io~projected/kube-api-access-tvqvn:{mountpoint:/var/lib/kubelet/pods/41bc42bb-f14c-4921-a5f4-5a7ceec653e6/volumes/kubernetes.io~projected/kube-api-access-tvqvn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51fd9e7b-409f-4896-b369-ca78469fc468/volumes/kubernetes.io~projected/kube-api-access-vhbrh:{mountpoint:/var/lib/kubelet/pods/51fd9e7b-409f-4896-b369-ca78469fc468/volumes/kubernetes.io~projected/kube-api-access-vhbrh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51fd9e7b-409f-4896-b369-ca78469fc468/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/51fd9e7b-409f-4896-b369-ca78469fc468/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volumes/kubernetes.io~projected/kube-api-access-5pnh8:{mountpoint:/var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volumes/kubernetes.io~projected/kube-api-access-5pnh8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb945a9-e793-4e53-bb13-38b3eb5022e4/volumes/kubernetes.io~projected/kube-api-access-88zsx:{mountpoint:/var/lib/kubelet/pods/7bb945a9-e793-4e53-bb13-38b3eb5022e4/volumes/kubernetes.io~projected/kube-api-access-88zsx major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a5332a6-581b-40bd-ba3c-284bcece1df1/volumes/kubernetes.io~projected/kube-api-access-dhtsz:{mountpoint:/var/lib/kubelet/pods/8a5332a6-581b-40bd-ba3c-284bcece1df1/volumes/kubernetes.io~projected/kube-api-access-dhtsz major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95a82d11-1674-4b89-bdeb-6fbf6caa5304/volumes/kubernetes.io~projected/kube-api-access-lmb7h:{mountpoint:/var/lib/kubelet/pods/95a82d11-1674-4b89-bdeb-6fbf6caa5304/volumes/kubernetes.io~projected/kube-api-access-lmb7h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbeed871-241a-4509-ac94-dbe5170904fe/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bbeed871-241a-4509-ac94-dbe5170904fe/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a49f7-9cf1-4bd1-aed4-c66384880090/volumes/kubernetes.io~projected/kube-api-access-nzzhb:{mountpoint:/var/lib/kubelet/pods/e36a49f7-9cf1-4bd1-aed4-c66384880090/volumes/kubernetes.io~projected/kube-api-access-nzzhb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f87a90af-8294-4303-b000-6b4e52289f79/volumes/kubernetes.io~projected/kube-api-access-4kwl6:{mountpoint:/var/lib/kubelet/pods/f87a90af-8294-4303-b000-6b4e52289f79/volumes/kubernetes.io~projected/kube-api-access-4kwl6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c/volumes/kubernetes.io~projected/kube-api-access-zd9bm:{mountpoint:/var/lib/kubelet/pods/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c/volumes/kubernetes.io~projected/kube-api-access-zd9bm major:0 minor:215 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ed5256d0d49afee65a2c32b641ac83b3e8926aee7c26f52d721bd5da0fb717e9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/53113b891c0432f3a35db093f4b8b52ebe39d4f7196b8b97893280eabb729074/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/30f0f923b94d5d85e9a2349351fac684960db672ff1e790b6392043c6cf80e02/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cefbc2f57fe7fddb2b88c157170965298ddefcd62e4b8124f62b5d06dde69174/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/66d7f44b9087b646423482568817c9a4f2aa83e87bd2e234e66a8af42eee7afd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/633e2962a6623a0b6b71a84449a6aeff24572ad7c9176c8f90f8ff9291cab96e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/76b9ae66f50a00a54e92c7b7ed5f04d82faae3ba814d997bdc9278db79365c57/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b2e7af9ca20aaf286591d64bdd7b4272e33314f315a5216c15631ada7469fc83/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2789e7b5f2833f5a7e9c0fd5207841bc686911618891945b6a912afc1893c51e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ddcb3ad755f12cba13bf436032106976e1d7d64466bd9567e509b5798c9f98c4/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/69ff0ad0ee61f4f9e901a0e187d2bd7f72d8907ed820836092bcf8b546c72d78/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/85ebae335a68886583c76db5bfe1cf82379b41ec134369ce275f34fe7c0dc211/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/512846b0eaf7157d382171144030c301f3104cbbbb757f296100636b5f65a5ad/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/54dd3118e55ba45747bc749c54ec1bf0e872c4e793f3039f3fa9fa347e735ed8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d5fdb7172c68ec4c6c20f616a4312c7207ed1b9070d1807e84dc705e5cc143e1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b2c57b3c6c6ace027b550dff94b57946333c294c6524ac189f740f2e1ddb19bd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4b6ba00c59f2a7a20e65c9de1a0c847f08ed960b108f76787ad7a7a32e9fa877/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c4bddfeb31d22b676cc077e1aa7321c60a5d390dd26345a9fc003aa28fbb9c2c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4bf33e0a1864d422693b16879bcd8e63f656838ac3547fba97b048c419428705/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f27f23f1bd8cb845b9e47329791a0ff95e459612fd3ed518963da2c14fbd3c30/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2836e669d640924540496bbec0c7033d8d1dd21bf7f9a4d4a8eb9a7575770cf7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/5e454fe3c6c51dfbac661251878bc279f0193287fe61c1b3a60567ed03fcd794/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/6ac7f69dc28248282fac88b7586604f47cb9b02887b25ea25c5da0a1f89ac166/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/98f33aa4f4de063c88d2a830071b5df7c9dc508a6982d4aed378af4096b9c1a6/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/e64500e90cc5107b51754ba69525533ce9dbaaaee711598ba64a6dada1ac6c38/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/38dd5504c16d6db3ca96b3bb548e65370b79b03b09f1e1cd523832c76c23b4ba/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/da26d37b9c155089ac95e5ca21ccf1f4cdcae9dde7a350e8f08ed215223980e3/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/1639895af933f3169eafca23a08541e9f968530f2e5b952ec2c07e0bfaf1c95a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/48008bd87b146ec1e24c5b6f6e9fbad6b62ffa6970e4d8d0b02a09792e70cedb/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9cc110d9d756a0ff370cef59ec1c0cf1be92c59d0a2621f680cc18428bfa9e8a/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/29f4436795c109f3fe43a933d2c495feec477cb1f7e3902a541994e6e765b777/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c0ea13c06fd4ddc375abfe31362b96026b87aa50cedadb4aecd54a1708636701/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/74ea497e2580babace27f08a3dbe3217a5d027af6ba700738994286e38c56b43/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/250efe033cb8c9ecd11772828cf2268933b9714d7c0553794a1f35d098f31a09/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/031daf39614613bacf4bca5dec9842006fed22b795db5b064193cd62bab3c9b3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/8bcba29af88172ecd457c6d04d25a6b1055fce24b684cb1c7b52a9afd507325c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/eb28d4f4f7d8f8c410899ad152c70149ee5f50521fff96c804e58190141c3f82/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/533407ecc8fd0ab54b26f9f829f6b83c47e6d30a0ef85a34942088a0b9fe7b70/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/38e753092d78586cef83ab6bb1651cb8f0de6afc976a73fa174ad3efc1ee1d66/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.112537    4451 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:20.111674435 +0000 UTC m=+0.054633131 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128982 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2218f61520bd412796ea8c12327573 SystemUUID:ec2218f6-1520-bd41-2796-ea8c12327573 BootID:fe01e702-c8d7-41e3-8fc8-6fac3f6585b6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volumes/kubernetes.io~projected/kube-api-access-5pnh8 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~projected/kube-api-access-bt7bd DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a49f7-9cf1-4bd1-aed4-c66384880090/volumes/kubernetes.io~projected/kube-api-access-nzzhb DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0295958e8bcb5a428a0a24b673881c0a6e83e1ed01241a164fb6203e03d143dc/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/41bc42bb-f14c-4921-a5f4-5a7ceec653e6/volumes/kubernetes.io~projected/kube-api-access-tvqvn DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17544f8b5b0d095ec599ccd3f6b164acf07ed4e8c2c94cf5439a063eeb6fef86/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a440fbded561f20cb4693babf63bf903842d77fd8b039c8fec31a21e01dee6c/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26757508-3500-4597-b243-ae17376bd037/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c6470dd-33de-42f4-a144-2fe004a117be/volumes/kubernetes.io~projected/kube-api-access-88r8z DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbeed871-241a-4509-ac94-dbe5170904fe/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51fd9e7b-409f-4896-b369-ca78469fc468/volumes/kubernetes.io~projected/kube-api-access-vhbrh DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6256ec4c420bf78ead6b2a4809ee0dcb66e113fdfca0c8168fb51264961564bb/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7520d944-da53-4455-868f-3880e5535345/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e49e136-156b-4dfa-a877-51b5c57d0ffa/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e8a4469121fe33a85cc0c15671c2a2c458ed9b8fa4c2152c7a859741754c31b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c/volumes/kubernetes.io~projected/kube-api-access-zd9bm DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51fd9e7b-409f-4896-b369-ca78469fc468/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5df049cdc4e855c5e831031b4e8f365f7d3ceabdbf9bdb54f5c5cc6eeb6b6bb9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/085095dd23072bccadc32aca79e255a93f043146cd3618ccdd4d8b4415087e0d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f87a90af-8294-4303-b000-6b4e52289f79/volumes/kubernetes.io~projected/kube-api-access-4kwl6 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40dc144e57f8b83a69f223241b66909c65c62fd9d00235e2b4618b1bc9cb6fad/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/811c8fc2b081fe531582c509457e1497ec0551e65e94355ff4f1015238c4421b/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49f1c7ce47274da324c27c2ccaf1630053509ad05c003e22d50a212fde5df20e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054/volumes/kubernetes.io~projected/kube-api-access-5hdhq DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb945a9-e793-4e53-bb13-38b3eb5022e4/volumes/kubernetes.io~projected/kube-api-access-88zsx DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a5332a6-581b-40bd-ba3c-284bcece1df1/volumes/kubernetes.io~projected/kube-api-access-dhtsz DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95a82d11-1674-4b89-bdeb-6fbf6caa5304/volumes/kubernetes.io~projected/kube-api-access-lmb7h DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/30bf6468-e2cc-4f55-a719-c33ab521bf43/volumes/kubernetes.io~projected/kube-api-access-ksbtf DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f921f4e2c5dca7f6437dd5bfbbcad4e539707f7ec8d2c6562ff63f590efbf09/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c3face65a42aea4dab555dbf90d93a829c23073f19655a1cbb34a3249a56766/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b6dd904-33a2-4497-9424-654062e29812/volumes/kubernetes.io~projected/kube-api-access-2vdhz DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0295958e8bcb5a4 MacAddress:fe:09:de:b2:fb:9f Speed:10000 Mtu:8901} {Name:085095dd23072bc MacAddress:72:f8:2b:14:a5:fc Speed:10000 Mtu:8901} {Name:5a440fbded561f2 MacAddress:26:02:63:50:b4:ac Speed:10000 Mtu:8901} {Name:811c8fc2b081fe5 MacAddress:16:81:d8:41:be:97 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d8:e7:b1:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:76:05:13:00:a8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d8:e7:b1:7b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c6:54:51:66:06:49 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:e6:94:39:9e:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.112772    4451 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.112815    4451 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113086    4451 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113104    4451 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113225    4451 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113233    4451 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113245    4451 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113259    4451 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113333    4451 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113515    4451 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113615    4451 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113649    4451 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113663    4451 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113673    4451 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113681    4451 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.113689    4451 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.114976    4451 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.114995    4451 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:14:20.116229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.115543    4451 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:14:20.117882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.117863    4451 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:14:20.117882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.117880    4451 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:14:20.118202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118002    4451 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:14:20.118202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118183    4451 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:14:20.118299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118210    4451 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:14:20.118299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118296    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118311    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118321    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118335    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118344    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118354    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118361    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118366    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118374    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:14:20.118379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118381    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:14:20.118799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118398    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:14:20.118799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118408    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:14:20.118799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118433    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:14:20.118799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.118442    4451 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:14:20.121789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.121773    4451 server.go:1263] "Started kubelet"
Oct 29 22:14:20.121874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.121841    4451 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:14:20.122316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.122258    4451 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:14:20.122423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.122336    4451 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:14:20.122338 ip-10-0-143-15 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:14:20.123154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.123133    4451 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:14:20.124045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124025    4451 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:14:20.124699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124684    4451 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:14:20.124790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124703    4451 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124856    4451 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:42 +0000 UTC" deadline="2028-08-09 06:39:53.720267674 +0000 UTC"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124893    4451 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15584h25m33.595378343s"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124941    4451 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.124950    4451 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:20.124992    4451 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-15.ec2.internal\" not found"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.125047    4451 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.125851    4451 factory.go:55] Registering systemd factory
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.125867    4451 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126586    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa11b25-01d3-4d6a-ad70-fcfe34ad324c" volumeName="kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126617    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054" volumeName="kubernetes.io/projected/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-kube-api-access-5hdhq" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126632    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6470dd-33de-42f4-a144-2fe004a117be" volumeName="kubernetes.io/projected/0c6470dd-33de-42f4-a144-2fe004a117be-kube-api-access-88r8z" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126645    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30bf6468-e2cc-4f55-a719-c33ab521bf43" volumeName="kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126660    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49e136-156b-4dfa-a877-51b5c57d0ffa" volumeName="kubernetes.io/secret/3e49e136-156b-4dfa-a877-51b5c57d0ffa-agent-certs" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126672    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a5332a6-581b-40bd-ba3c-284bcece1df1" volumeName="kubernetes.io/projected/8a5332a6-581b-40bd-ba3c-284bcece1df1-kube-api-access-dhtsz" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126685    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb945a9-e793-4e53-bb13-38b3eb5022e4" volumeName="kubernetes.io/projected/7bb945a9-e793-4e53-bb13-38b3eb5022e4-kube-api-access-88zsx" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126697    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26757508-3500-4597-b243-ae17376bd037" volumeName="kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-tmp" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126713    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fd9e7b-409f-4896-b369-ca78469fc468" volumeName="kubernetes.io/projected/51fd9e7b-409f-4896-b369-ca78469fc468-kube-api-access-vhbrh" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126725    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fd9e7b-409f-4896-b369-ca78469fc468" volumeName="kubernetes.io/secret/51fd9e7b-409f-4896-b369-ca78469fc468-cert" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126738    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7520d944-da53-4455-868f-3880e5535345" volumeName="kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-config" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126745    4451 factory.go:153] Registering CRI-O factory
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126752    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7520d944-da53-4455-868f-3880e5535345" volumeName="kubernetes.io/projected/7520d944-da53-4455-868f-3880e5535345-kube-api-access-5pnh8" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126761    4451 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126765    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f87a90af-8294-4303-b000-6b4e52289f79" volumeName="kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126778    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6dd904-33a2-4497-9424-654062e29812" volumeName="kubernetes.io/empty-dir/1b6dd904-33a2-4497-9424-654062e29812-data-volume" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126794    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26757508-3500-4597-b243-ae17376bd037" volumeName="kubernetes.io/projected/26757508-3500-4597-b243-ae17376bd037-kube-api-access-bt7bd" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126807    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41bc42bb-f14c-4921-a5f4-5a7ceec653e6" volumeName="kubernetes.io/projected/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-kube-api-access-tvqvn" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126813    4451 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126818    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7520d944-da53-4455-868f-3880e5535345" volumeName="kubernetes.io/secret/7520d944-da53-4455-868f-3880e5535345-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126832    4451 factory.go:103] Registering Raw factory
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126847    4451 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126833    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054" volumeName="kubernetes.io/configmap/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-serviceca" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126864    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30bf6468-e2cc-4f55-a719-c33ab521bf43" volumeName="kubernetes.io/projected/30bf6468-e2cc-4f55-a719-c33ab521bf43-kube-api-access-ksbtf" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126875    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb945a9-e793-4e53-bb13-38b3eb5022e4" volumeName="kubernetes.io/configmap/7bb945a9-e793-4e53-bb13-38b3eb5022e4-config-volume" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126888    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb945a9-e793-4e53-bb13-38b3eb5022e4" volumeName="kubernetes.io/empty-dir/7bb945a9-e793-4e53-bb13-38b3eb5022e4-tmp-dir" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126900    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7520d944-da53-4455-868f-3880e5535345" volumeName="kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126929    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7520d944-da53-4455-868f-3880e5535345" volumeName="kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-env-overrides" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126942    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a49f7-9cf1-4bd1-aed4-c66384880090" volumeName="kubernetes.io/projected/e36a49f7-9cf1-4bd1-aed4-c66384880090-kube-api-access-nzzhb" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126955    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26757508-3500-4597-b243-ae17376bd037" volumeName="kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-etc-tuned" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126967    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30bf6468-e2cc-4f55-a719-c33ab521bf43" volumeName="kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126981    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41bc42bb-f14c-4921-a5f4-5a7ceec653e6" volumeName="kubernetes.io/empty-dir/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-tmp-dir" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.126993    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbeed871-241a-4509-ac94-dbe5170904fe" volumeName="kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127012    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f87a90af-8294-4303-b000-6b4e52289f79" volumeName="kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127027    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f87a90af-8294-4303-b000-6b4e52289f79" volumeName="kubernetes.io/projected/f87a90af-8294-4303-b000-6b4e52289f79-kube-api-access-4kwl6" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127040    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6dd904-33a2-4497-9424-654062e29812" volumeName="kubernetes.io/projected/1b6dd904-33a2-4497-9424-654062e29812-kube-api-access-2vdhz" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127053    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30bf6468-e2cc-4f55-a719-c33ab521bf43" volumeName="kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127067    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49e136-156b-4dfa-a877-51b5c57d0ffa" volumeName="kubernetes.io/configmap/3e49e136-156b-4dfa-a877-51b5c57d0ffa-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127091    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95a82d11-1674-4b89-bdeb-6fbf6caa5304" volumeName="kubernetes.io/configmap/95a82d11-1674-4b89-bdeb-6fbf6caa5304-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127103    4451 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95a82d11-1674-4b89-bdeb-6fbf6caa5304" volumeName="kubernetes.io/projected/95a82d11-1674-4b89-bdeb-6fbf6caa5304-kube-api-access-lmb7h" seLinuxMountContext=""
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127115    4451 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127123    4451 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:14:20.127438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.127234    4451 manager.go:319] Starting recovery of all containers
Oct 29 22:14:20.153324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.153269    4451 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:14:20.155136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.155100    4451 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:14:20.155283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.155271    4451 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:14:20.155379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.155371    4451 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:14:20.155489 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:20.155471    4451 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:14:20.170772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.170749    4451 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:14:20.182613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.182583    4451 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="5fae997e117515b904b16fa38fe02e7e1a56aed517611a6043353f24bb78c1df" exitCode=0
Oct 29 22:14:20.182613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.182612    4451 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="ec961f153bb20754761e6ef61c91d5d398428e096ed997751036daed8fabca45" exitCode=0
Oct 29 22:14:20.182791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.182623    4451 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="3987d4d5ffe75938181051b92233774d9bc82f45213d7873d022c772c4953739" exitCode=0
Oct 29 22:14:20.182791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.182631    4451 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="173eb099487ce5887ee4b3467561d6bf9ad6dc109b364d898398cae44a01c93e" exitCode=0
Oct 29 22:14:20.182791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.182639    4451 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="16472ba9e480d887a154573e6a8f733d676fd268723c2447256cc28c833c6fb0" exitCode=0
Oct 29 22:14:20.182791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.182646    4451 generic.go:356] "Generic (PLEG): container finished" podID="30bf6468-e2cc-4f55-a719-c33ab521bf43" containerID="56eae3f7161cb32f806bae5ebe7a5ef19aa24a3c3f90811ecc19b8c8049ec54b" exitCode=0
Oct 29 22:14:20.187555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.187530    4451 generic.go:356] "Generic (PLEG): container finished" podID="e06e61f905fbd53de8ba68af88289e72" containerID="dd523e9c2b668fe002fd40ad34b03e5cc284a95ad664e6d6e709a4aa2a5c04e9" exitCode=0
Oct 29 22:14:20.193253 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.193230    4451 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:20.193253 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.193247    4451 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:20.193466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.193449    4451 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:20.193466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.193460    4451 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:20.195310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.195292    4451 manager.go:324] Recovery completed
Oct 29 22:14:20.210452 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210435    4451 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:14:20.210452 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210449    4451 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:14:20.210570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210467    4451 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:14:20.210613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210599    4451 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:14:20.210647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210614    4451 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:14:20.210647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210630    4451 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:14:20.210647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210643    4451 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:14:20.210647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210648    4451 policy_none.go:50] "Start"
Oct 29 22:14:20.210751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210660    4451 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:14:20.210751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210670    4451 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:14:20.210804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210784    4451 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:14:20.210804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210791    4451 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:14:20.210804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.210796    4451 policy_none.go:44] "Start"
Oct 29 22:14:20.214613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.214589    4451 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:14:20.217131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.217120    4451 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:14:20.217580 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:20.217141    4451 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:14:20.217617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.217583    4451 server.go:87] "Starting device plugin registration server"
Oct 29 22:14:20.217821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.217812    4451 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:14:20.217851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.217822    4451 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:14:20.217951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.217934    4451 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:14:20.218063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.218031    4451 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:14:20.218134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.218071    4451 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:14:20.218134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.218078    4451 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:14:20.218245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.218111    4451 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:20.218061398Z","UUID":"c355b1e8-e277-49ca-9514-29abfe705b6d","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:14:20.219216 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:20.219199    4451 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:14:20.219424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.219378    4451 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:14:20.219424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.219412    4451 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:14:20.256358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256320    4451 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"]
Oct 29 22:14:20.256804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256762    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" event={"ID":"b64de3cb655c6c434e0a94257c87f3e3","Type":"ContainerStarted","Data":"15d0b2c5023eb2b29fe32bbd41dc3a83c45866f0b399f6c3e80bbb0e72d4e70d"}
Oct 29 22:14:20.256893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256809    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal" event={"ID":"b64de3cb655c6c434e0a94257c87f3e3","Type":"ContainerStarted","Data":"f2b4eccaba47a857f1ba436c60ba3def67fd10ddf87f2327edd617eeb3245616"}
Oct 29 22:14:20.256893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256855    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"e06e61f905fbd53de8ba68af88289e72","Type":"ContainerStarted","Data":"058eab6bb71464574b236721040646b869a81791c49a8f5c4ad4bf3e9ab1c04d"}
Oct 29 22:14:20.256893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256867    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"e06e61f905fbd53de8ba68af88289e72","Type":"ContainerDied","Data":"dd523e9c2b668fe002fd40ad34b03e5cc284a95ad664e6d6e709a4aa2a5c04e9"}
Oct 29 22:14:20.256893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256878    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal" event={"ID":"e06e61f905fbd53de8ba68af88289e72","Type":"ContainerStarted","Data":"6a33403b0b7d48e232cdf57bb3f679dd3c4f4ead73543b4ec63da7409b4bf170"}
Oct 29 22:14:20.257052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256959    4451 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.257052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.256992    4451 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.276740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.276720    4451 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:20.276829 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:20.276754    4451 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.277218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.277205    4451 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:20.277259 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:20.277235    4451 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.318364 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.318342    4451 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:14:20.319598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.319583    4451 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:14:20.319660 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.319609    4451 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:14:20.319660 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.319620    4451 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-15.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:14:20.319660 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.319652    4451 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.357231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.357207    4451 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.357328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.357264    4451 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.372642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.372614    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b64de3cb655c6c434e0a94257c87f3e3-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"b64de3cb655c6c434e0a94257c87f3e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.372734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.372647    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.372734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.372671    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.473532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.473451    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.473532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.473478    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.473532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.473507    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.473735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.473538    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b64de3cb655c6c434e0a94257c87f3e3-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"b64de3cb655c6c434e0a94257c87f3e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.473735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.473585    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b64de3cb655c6c434e0a94257c87f3e3-config\") pod \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" (UID: \"b64de3cb655c6c434e0a94257c87f3e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:20.473735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:20.473609    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e06e61f905fbd53de8ba68af88289e72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" (UID: \"e06e61f905fbd53de8ba68af88289e72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:21.114808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.114779    4451 apiserver.go:52] "Watching apiserver"
Oct 29 22:14:21.189815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.189789    4451 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:21.189963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.189948    4451 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:21.198621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.198605    4451 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:14:21.199024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.199006    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-g7bbj","kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal","openshift-dns/dns-default-2lqth","openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj","openshift-multus/multus-additional-cni-plugins-hnw4v","openshift-ovn-kubernetes/ovnkube-node-8mh74","openshift-ingress-canary/ingress-canary-nftxr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal","openshift-multus/multus-ngnk6","openshift-insights/insights-runtime-extractor-qvpbn","openshift-dns/node-resolver-l7cws","kube-system/global-pull-secret-syncer-j9lz9","openshift-network-diagnostics/network-check-target-qp2mt","openshift-network-operator/iptables-alerter-wx7df","openshift-cluster-node-tuning-operator/tuned-7nhrb","kube-system/konnectivity-agent-x4bzm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x","openshift-image-registry/node-ca-5q62r"]
Oct 29 22:14:21.204334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.204314    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:21.205578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.205560    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.205974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.205958    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.209121 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.209104    4451 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:21.209213 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.209156    4451 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-15.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:21.211975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.211957    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.211975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.211964    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:14:21.212092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212017    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:14:21.212092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212078    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.212440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212406    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.212440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212437    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p5dkc\""
Oct 29 22:14:21.212590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212466    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:14:21.212590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212469    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr68l\""
Oct 29 22:14:21.212590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212479    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.212590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212534    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwjl8\""
Oct 29 22:14:21.212590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212571    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.212808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212693    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.212808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212705    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:14:21.212808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212708    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.212981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.212965    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.213049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.213015    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjwp\""
Oct 29 22:14:21.216224 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.216211    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:14:21.216921 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.216896    4451 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:21.216975 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.216952    4451 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-15.ec2.internal"
Oct 29 22:14:21.226064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.226052    4451 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:14:21.277245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277224    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-config\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.277334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277246    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7520d944-da53-4455-868f-3880e5535345-ovn-node-metrics-cert\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.277334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277266    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-socket-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.277334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277284    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zsx\" (UniqueName: \"kubernetes.io/projected/7bb945a9-e793-4e53-bb13-38b3eb5022e4-kube-api-access-88zsx\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.277334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277303    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-cni-binary-copy\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.277527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277394    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-multus-daemon-config\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.277527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277423    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-serviceca\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.277527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277446    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-host\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.277527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277461    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51fd9e7b-409f-4896-b369-ca78469fc468-cert\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:21.277527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277482    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-kubelet\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.277763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277677    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-serviceca\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.277902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277882    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:21.277971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277935    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-cnibin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.278032 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.277979    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwl6\" (UniqueName: \"kubernetes.io/projected/f87a90af-8294-4303-b000-6b4e52289f79-kube-api-access-4kwl6\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.278078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278038    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-host\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.278133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278108    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-multus-daemon-config\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.278182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278124    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f87a90af-8294-4303-b000-6b4e52289f79-cni-binary-copy\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.278182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278151    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-hosts-file\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.278272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278181    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysconfig\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.278272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278208    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-systemd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.278272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278234    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-multus-certs\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.278350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278270    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.278350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278297    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e49e136-156b-4dfa-a877-51b5c57d0ffa-konnectivity-ca\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:21.278350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278314    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-etc-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.278350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278331    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.278507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278353    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-env-overrides\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.278507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278483    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-binary-copy\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.278813 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278798    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-registration-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.278843 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278821    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-sys-fs\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.278843 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278838    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e49e136-156b-4dfa-a877-51b5c57d0ffa-agent-certs\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:21.279058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278856    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r8z\" (UniqueName: \"kubernetes.io/projected/0c6470dd-33de-42f4-a144-2fe004a117be-kube-api-access-88r8z\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.279058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278976    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-system-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.279058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.278995    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-etc-kubernetes\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.279058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279013    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdhq\" (UniqueName: \"kubernetes.io/projected/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-kube-api-access-5hdhq\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.279058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279057    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-modprobe-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.279271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279086    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.279271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279102    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-bin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.279271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279122    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-sys\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.279271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279158    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbrh\" (UniqueName: \"kubernetes.io/projected/51fd9e7b-409f-4896-b369-ca78469fc468-kube-api-access-vhbrh\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:21.279424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279278    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-slash\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.279424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279310    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-var-lib-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.279424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279348    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6dd904-33a2-4497-9424-654062e29812-data-volume\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.279424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279377    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-kubelet\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.279424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279392    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-var-lib-kubelet\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.279424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279408    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb945a9-e793-4e53-bb13-38b3eb5022e4-tmp-dir\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279448    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-cnibin\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279467    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279473    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6dd904-33a2-4497-9424-654062e29812-data-volume\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279483    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-ovn\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279540    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb945a9-e793-4e53-bb13-38b3eb5022e4-tmp-dir\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279565    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnh8\" (UniqueName: \"kubernetes.io/projected/7520d944-da53-4455-868f-3880e5535345-kube-api-access-5pnh8\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.279614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279610    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6dd904-33a2-4497-9424-654062e29812-crio-socket\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279627    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95a82d11-1674-4b89-bdeb-6fbf6caa5304-host-slash\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279654    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-tmp-dir\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279674    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-device-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279700    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb945a9-e793-4e53-bb13-38b3eb5022e4-config-volume\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279741    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279796    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-tmp-dir\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279823    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbtf\" (UniqueName: \"kubernetes.io/projected/30bf6468-e2cc-4f55-a719-c33ab521bf43-kube-api-access-ksbtf\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279869    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-lib-modules\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.279895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279894    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279935    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279953    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-netns\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.279972    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7bd\" (UniqueName: \"kubernetes.io/projected/26757508-3500-4597-b243-ae17376bd037-kube-api-access-bt7bd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280018    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-netns\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280035    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-log-socket\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280056    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-script-lib\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280086    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280125    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzhb\" (UniqueName: \"kubernetes.io/projected/e36a49f7-9cf1-4bd1-aed4-c66384880090-kube-api-access-nzzhb\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280150    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-dbus\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280208    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-system-cni-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280235    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-etc-selinux\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280258    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-kubernetes\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280284    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-tmp\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280350    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdhz\" (UniqueName: \"kubernetes.io/projected/1b6dd904-33a2-4497-9424-654062e29812-kube-api-access-2vdhz\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280374    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-tmp\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.280384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280383    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-kubelet-config\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280417    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-socket-dir-parent\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280451    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-conf-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280470    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmb7h\" (UniqueName: \"kubernetes.io/projected/95a82d11-1674-4b89-bdeb-6fbf6caa5304-kube-api-access-lmb7h\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280484    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280502    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-systemd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280516    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-node-log\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280531    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-netd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280566    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280606    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280643    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280680    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-k8s-cni-cncf-io\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280724    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-multus\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280756    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-hostroot\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280780    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280801    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-run\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280845    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95a82d11-1674-4b89-bdeb-6fbf6caa5304-iptables-alerter-script\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280865    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-etc-tuned\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280880    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-systemd-units\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280894    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-bin\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280925    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280943    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtsz\" (UniqueName: \"kubernetes.io/projected/8a5332a6-581b-40bd-ba3c-284bcece1df1-kube-api-access-dhtsz\") pod \"migrator-66df5d496b-4jbpj\" (UID: \"8a5332a6-581b-40bd-ba3c-284bcece1df1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.280964    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26757508-3500-4597-b243-ae17376bd037-etc-tuned\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.281018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.281017    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqvn\" (UniqueName: \"kubernetes.io/projected/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-kube-api-access-tvqvn\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.281757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.281110    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-conf\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.281757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.281128    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.281757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.281153    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.297832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.297814    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r8z\" (UniqueName: \"kubernetes.io/projected/0c6470dd-33de-42f4-a144-2fe004a117be-kube-api-access-88r8z\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.298112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.298096    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbtf\" (UniqueName: \"kubernetes.io/projected/30bf6468-e2cc-4f55-a719-c33ab521bf43-kube-api-access-ksbtf\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.298687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.298671    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdhq\" (UniqueName: \"kubernetes.io/projected/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-kube-api-access-5hdhq\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.299275 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.299256    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzhb\" (UniqueName: \"kubernetes.io/projected/e36a49f7-9cf1-4bd1-aed4-c66384880090-kube-api-access-nzzhb\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:21.299354 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.299336    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqvn\" (UniqueName: \"kubernetes.io/projected/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-kube-api-access-tvqvn\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.300940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.300903    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zsx\" (UniqueName: \"kubernetes.io/projected/7bb945a9-e793-4e53-bb13-38b3eb5022e4-kube-api-access-88zsx\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.301261 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.301243    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwl6\" (UniqueName: \"kubernetes.io/projected/f87a90af-8294-4303-b000-6b4e52289f79-kube-api-access-4kwl6\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.312405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312388    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.312492 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312473    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.312551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312505    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.312551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312492    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:14:21.312551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312514    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:14:21.312551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312536    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.312699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.312601    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:14:21.313726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.313709    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.313819 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.313805    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:14:21.313966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.313952    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.314002 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.313966    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.314047 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314027    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:14:21.314258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314243    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.314321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314297    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:14:21.314321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314306    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.314411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314308    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.314411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314403    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:14:21.314481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314414    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:14:21.314710 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314696    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.314756 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.314745    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.315479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.315464    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:21.315479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.315474    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:14:21.315650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.315638    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:14:21.315967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.315951    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fz5pc\""
Oct 29 22:14:21.317293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.317278    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnh8\" (UniqueName: \"kubernetes.io/projected/7520d944-da53-4455-868f-3880e5535345-kube-api-access-5pnh8\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.318171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318158    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7520d944-da53-4455-868f-3880e5535345-ovn-node-metrics-cert\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.318245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318230    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-config\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.318544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318523    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtsz\" (UniqueName: \"kubernetes.io/projected/8a5332a6-581b-40bd-ba3c-284bcece1df1-kube-api-access-dhtsz\") pod \"migrator-66df5d496b-4jbpj\" (UID: \"8a5332a6-581b-40bd-ba3c-284bcece1df1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4jbpj"
Oct 29 22:14:21.318764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318748    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-env-overrides\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.318764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318760    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e49e136-156b-4dfa-a877-51b5c57d0ffa-konnectivity-ca\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:21.318890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318747    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9bm\" (UniqueName: \"kubernetes.io/projected/ffa11b25-01d3-4d6a-ad70-fcfe34ad324c-kube-api-access-zd9bm\") pod \"network-check-target-qp2mt\" (UID: \"ffa11b25-01d3-4d6a-ad70-fcfe34ad324c\") " pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:21.318942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318902    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdhz\" (UniqueName: \"kubernetes.io/projected/1b6dd904-33a2-4497-9424-654062e29812-kube-api-access-2vdhz\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.318973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.318952    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmb7h\" (UniqueName: \"kubernetes.io/projected/95a82d11-1674-4b89-bdeb-6fbf6caa5304-kube-api-access-lmb7h\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.319411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.319397    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbrh\" (UniqueName: \"kubernetes.io/projected/51fd9e7b-409f-4896-b369-ca78469fc468-kube-api-access-vhbrh\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:21.320453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.320439    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb945a9-e793-4e53-bb13-38b3eb5022e4-config-volume\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.320532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.320519    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7520d944-da53-4455-868f-3880e5535345-ovnkube-script-lib\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.320574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.320562    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30bf6468-e2cc-4f55-a719-c33ab521bf43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.321645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.321633    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95a82d11-1674-4b89-bdeb-6fbf6caa5304-iptables-alerter-script\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.321845 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.321825    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v7npq\""
Oct 29 22:14:21.321954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.321844    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-977wj\""
Oct 29 22:14:21.322322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.322298    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7vbxc\""
Oct 29 22:14:21.324726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324706    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:14:21.324872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324712    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xsn6t\""
Oct 29 22:14:21.324872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324715    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:14:21.325009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324873    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpwxr\""
Oct 29 22:14:21.325009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324726    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f584j\""
Oct 29 22:14:21.325009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324743    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:14:21.325193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324767    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cdnlg\""
Oct 29 22:14:21.325225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324777    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:14:21.325225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324776    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nqlz\""
Oct 29 22:14:21.325225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324795    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwm9d\""
Oct 29 22:14:21.325306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324800    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:14:21.325306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.325226    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:21.325306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.324718    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjh7c\""
Oct 29 22:14:21.328335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.328316    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51fd9e7b-409f-4896-b369-ca78469fc468-cert\") pod \"ingress-canary-nftxr\" (UID: \"51fd9e7b-409f-4896-b369-ca78469fc468\") " pod="openshift-ingress-canary/ingress-canary-nftxr"
Oct 29 22:14:21.328570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.328516    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7bd\" (UniqueName: \"kubernetes.io/projected/26757508-3500-4597-b243-ae17376bd037-kube-api-access-bt7bd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.328660 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.328642    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kj4hx\""
Oct 29 22:14:21.329803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.329787    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e49e136-156b-4dfa-a877-51b5c57d0ffa-agent-certs\") pod \"konnectivity-agent-x4bzm\" (UID: \"3e49e136-156b-4dfa-a877-51b5c57d0ffa\") " pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:21.330892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.330879    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbeed871-241a-4509-ac94-dbe5170904fe-original-pull-secret\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.381997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.381942    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-bin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.381997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.381970    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-sys\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.381995    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-slash\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382017    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-var-lib-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382058    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-bin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382069    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-var-lib-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382070    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-sys\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382071    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-kubelet\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382092    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-slash\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382109    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-var-lib-kubelet\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.382123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382111    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-kubelet\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382136    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-var-lib-kubelet\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382143    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-cnibin\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382161    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-ovn\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382175    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6dd904-33a2-4497-9424-654062e29812-crio-socket\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382189    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95a82d11-1674-4b89-bdeb-6fbf6caa5304-host-slash\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382204    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-device-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382219    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-lib-modules\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382243    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-cnibin\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382247    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-ovn\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382256    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95a82d11-1674-4b89-bdeb-6fbf6caa5304-host-slash\") pod \"iptables-alerter-wx7df\" (UID: \"95a82d11-1674-4b89-bdeb-6fbf6caa5304\") " pod="openshift-network-operator/iptables-alerter-wx7df"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382272    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-device-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382278    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382288    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6dd904-33a2-4497-9424-654062e29812-crio-socket\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382303    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-netns\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382318    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-lib-modules\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382324    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-netns\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382338    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-log-socket\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382347    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-netns\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382354    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382371    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-netns\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382383    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-log-socket\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.382385    4451 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382402    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-dbus\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382429    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-system-cni-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382457    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-etc-selinux\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382473    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-dbus\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.382468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382481    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-kubernetes\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.382501    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:21.882481186 +0000 UTC m=+1.825439892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382512    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-system-cni-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382532    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-kubernetes\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382545    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-etc-selinux\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382549    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-kubelet-config\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382581    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-socket-dir-parent\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382597    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-conf-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382617    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382615    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbeed871-241a-4509-ac94-dbe5170904fe-kubelet-config\") pod \"global-pull-secret-syncer-j9lz9\" (UID: \"bbeed871-241a-4509-ac94-dbe5170904fe\") " pod="kube-system/global-pull-secret-syncer-j9lz9"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382650    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-conf-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382659    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-systemd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382671    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-socket-dir-parent\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382696    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-node-log\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382694    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-systemd\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382723    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382727    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-netd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382757    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-netd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382763    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-node-log\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382788    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382786    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6dd904-33a2-4497-9424-654062e29812-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382817    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382831    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382839    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-k8s-cni-cncf-io\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382854    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-multus\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382869    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-hostroot\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382885    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382907    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-k8s-cni-cncf-io\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382932    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-run\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382944    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-hostroot\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382955    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30bf6468-e2cc-4f55-a719-c33ab521bf43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hnw4v\" (UID: \"30bf6468-e2cc-4f55-a719-c33ab521bf43\") " pod="openshift-multus/multus-additional-cni-plugins-hnw4v"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382960    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-systemd-units\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382972    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-multus-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382976    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-run\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382983    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-bin\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.382991    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-var-lib-cni-multus\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383009    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-systemd-units\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383041    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-cni-bin\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383069    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-conf\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383094    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383117    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383153    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383160    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383173    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-socket-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383187    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysctl-conf\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383191    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-host\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383222    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-host\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383231    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-kubelet\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383254    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383268    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-cnibin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383268    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-kubelet\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383281    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-host\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383295    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-hosts-file\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383315    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysconfig\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383321    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-cnibin\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383234    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-socket-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383335    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054-host\") pod \"node-ca-5q62r\" (UID: \"0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054\") " pod="openshift-image-registry/node-ca-5q62r"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383338    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-systemd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.383342    4451 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383352    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41bc42bb-f14c-4921-a5f4-5a7ceec653e6-hosts-file\") pod \"node-resolver-l7cws\" (UID: \"41bc42bb-f14c-4921-a5f4-5a7ceec653e6\") " pod="openshift-dns/node-resolver-l7cws"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383365    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-sysconfig\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383367    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-multus-certs\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383371    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-run-systemd\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.383397    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:21.883381924 +0000 UTC m=+1.826340622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : secret "metrics-daemon-secret" not found
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383407    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-host-run-multus-certs\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383415    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-etc-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383429    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-etc-openvswitch\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383442    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383468    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-registration-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383491    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-sys-fs\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383499    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7520d944-da53-4455-868f-3880e5535345-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mh74\" (UID: \"7520d944-da53-4455-868f-3880e5535345\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383514    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-registration-dir\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383514    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-system-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383537    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-etc-kubernetes\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383543    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c6470dd-33de-42f4-a144-2fe004a117be-sys-fs\") pod \"aws-ebs-csi-driver-node-lc88x\" (UID: \"0c6470dd-33de-42f4-a144-2fe004a117be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lc88x"
Oct 29 22:14:21.383498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383553    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-modprobe-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.385889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383564    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-system-cni-dir\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.385889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383571    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f87a90af-8294-4303-b000-6b4e52289f79-etc-kubernetes\") pod \"multus-ngnk6\" (UID: \"f87a90af-8294-4303-b000-6b4e52289f79\") " pod="openshift-multus/multus-ngnk6"
Oct 29 22:14:21.385889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383568    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.385889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383638    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26757508-3500-4597-b243-ae17376bd037-etc-modprobe-d\") pod \"tuned-7nhrb\" (UID: \"26757508-3500-4597-b243-ae17376bd037\") " pod="openshift-cluster-node-tuning-operator/tuned-7nhrb"
Oct 29 22:14:21.385889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.383883    4451 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:14:21.385889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.385811    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6dd904-33a2-4497-9424-654062e29812-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvpbn\" (UID: \"1b6dd904-33a2-4497-9424-654062e29812\") " pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.507052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.507024    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvpbn"
Oct 29 22:14:21.831278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.831244    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qvpbn"]
Oct 29 22:14:21.834846 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:21.834814    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6dd904_33a2_4497_9424_654062e29812.slice/crio-83fbb4332221ad1aeb26678c09e6cdb8ba64e73f769d005ee20efb5d9b5d959b WatchSource:0}: Error finding container 83fbb4332221ad1aeb26678c09e6cdb8ba64e73f769d005ee20efb5d9b5d959b: Status 404 returned error can't find the container with id 83fbb4332221ad1aeb26678c09e6cdb8ba64e73f769d005ee20efb5d9b5d959b
Oct 29 22:14:21.871719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.871700    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7cws_41bc42bb-f14c-4921-a5f4-5a7ceec653e6/dns-node-resolver/0.log"
Oct 29 22:14:21.887027 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.887005    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:21.887099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.887056    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:21.887157 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.887139    4451 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:21.887189 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.887154    4451 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:21.887217 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.887194    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.887181777 +0000 UTC m=+2.830140460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : secret "metrics-daemon-secret" not found
Oct 29 22:14:21.887217 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:21.887207    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.887201779 +0000 UTC m=+2.830160462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:21.920099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:21.920082    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:14:22.192743 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:22.192714    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvpbn" event={"ID":"1b6dd904-33a2-4497-9424-654062e29812","Type":"ContainerStarted","Data":"b77a050a526117be65d46ee7b4e0df08071805030760c749fdd5de43f27e5812"}
Oct 29 22:14:22.193119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:22.192769    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvpbn" event={"ID":"1b6dd904-33a2-4497-9424-654062e29812","Type":"ContainerStarted","Data":"83fbb4332221ad1aeb26678c09e6cdb8ba64e73f769d005ee20efb5d9b5d959b"}
Oct 29 22:14:22.384652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:22.384615    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:22.672161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:22.672139    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5q62r_0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054/node-ca/0.log"
Oct 29 22:14:22.893131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:22.893097    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:22.893252 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:22.893153    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:22.893252 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:22.893239    4451 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.893312 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:22.893257    4451 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:22.893312 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:22.893294    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:24.893280011 +0000 UTC m=+4.836238694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.893312 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:22.893308    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs podName:e36a49f7-9cf1-4bd1-aed4-c66384880090 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:24.893302404 +0000 UTC m=+4.836261088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs") pod "network-metrics-daemon-g7bbj" (UID: "e36a49f7-9cf1-4bd1-aed4-c66384880090") : secret "metrics-daemon-secret" not found
Oct 29 22:14:23.197118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:23.197046    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvpbn" event={"ID":"1b6dd904-33a2-4497-9424-654062e29812","Type":"ContainerStarted","Data":"33b0390ded83a9acf795956b9a18e319a2d997c11c2173d05b7bd9cf8db4bf54"}
Oct 29 22:14:23.871466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:23.871428    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nftxr_51fd9e7b-409f-4896-b369-ca78469fc468/serve-healthcheck-canary/0.log"
Oct 29 22:14:24.071729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:24.071692    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4jbpj_8a5332a6-581b-40bd-ba3c-284bcece1df1/migrator/0.log"
Oct 29 22:14:24.201346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:24.201275    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvpbn" event={"ID":"1b6dd904-33a2-4497-9424-654062e29812","Type":"ContainerStarted","Data":"b44a4d8399d6126b714bc2454e2c224ee4d94f279dcedb355006bda1ee80cec6"}
Oct 29 22:14:24.905273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:24.905229    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:24.905445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:24.905292    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:24.905445 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:24.905380    4451 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:24.905445 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:24.905430    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls podName:7bb945a9-e793-4e53-bb13-38b3eb5022e4 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:28.905418766 +0000 UTC m=+8.848377449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls") pod "dns-default-2lqth" (UID: "7bb945a9-e793-4e53-bb13-38b3eb5022e4") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:24.907627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:24.907599    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e36a49f7-9cf1-4bd1-aed4-c66384880090-metrics-certs\") pod \"network-metrics-daemon-g7bbj\" (UID: \"e36a49f7-9cf1-4bd1-aed4-c66384880090\") " pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:25.104996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:25.104965    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7bbj"
Oct 29 22:14:25.385643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:25.385615    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:25.393756 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:25.393734    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g7bbj"]
Oct 29 22:14:25.397080 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:25.397058    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36a49f7_9cf1_4bd1_aed4_c66384880090.slice/crio-be862fc837729e8da1bb13f530549f2fe7a6dd31dbeb3aee63812243ea5a9960 WatchSource:0}: Error finding container be862fc837729e8da1bb13f530549f2fe7a6dd31dbeb3aee63812243ea5a9960: Status 404 returned error can't find the container with id be862fc837729e8da1bb13f530549f2fe7a6dd31dbeb3aee63812243ea5a9960
Oct 29 22:14:26.208713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:26.208682    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7bbj" event={"ID":"e36a49f7-9cf1-4bd1-aed4-c66384880090","Type":"ContainerStarted","Data":"be862fc837729e8da1bb13f530549f2fe7a6dd31dbeb3aee63812243ea5a9960"}
Oct 29 22:14:27.213564 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:27.213527    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7bbj" event={"ID":"e36a49f7-9cf1-4bd1-aed4-c66384880090","Type":"ContainerStarted","Data":"117694e7c58999d02ce0424f1ed166fb3dd80913c6c5eb20863df7e988997c30"}
Oct 29 22:14:27.213976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:27.213603    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7bbj" event={"ID":"e36a49f7-9cf1-4bd1-aed4-c66384880090","Type":"ContainerStarted","Data":"4696a8236d7d8d274b5c47cdbc813046a970a4dd6b4083f36c1fb1e2e2ad256d"}
Oct 29 22:14:27.816042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:27.816013    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:27.816261 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:27.816247    4451 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:14:28.931549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:28.931513    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:28.933751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:28.933727    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bb945a9-e793-4e53-bb13-38b3eb5022e4-metrics-tls\") pod \"dns-default-2lqth\" (UID: \"7bb945a9-e793-4e53-bb13-38b3eb5022e4\") " pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:29.006789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:29.006750    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:29.292429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:29.292353    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2lqth"]
Oct 29 22:14:29.295629 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:29.295603    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb945a9_e793_4e53_bb13_38b3eb5022e4.slice/crio-86f3d1ff9c3d50c663bc22a6feb02325551a9a30590f8b2094ddf3415f2c5bc9 WatchSource:0}: Error finding container 86f3d1ff9c3d50c663bc22a6feb02325551a9a30590f8b2094ddf3415f2c5bc9: Status 404 returned error can't find the container with id 86f3d1ff9c3d50c663bc22a6feb02325551a9a30590f8b2094ddf3415f2c5bc9
Oct 29 22:14:30.223807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:30.223771    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2lqth" event={"ID":"7bb945a9-e793-4e53-bb13-38b3eb5022e4","Type":"ContainerStarted","Data":"86f3d1ff9c3d50c663bc22a6feb02325551a9a30590f8b2094ddf3415f2c5bc9"}
Oct 29 22:14:31.229150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:31.229116    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2lqth" event={"ID":"7bb945a9-e793-4e53-bb13-38b3eb5022e4","Type":"ContainerStarted","Data":"a5f203d88903001bf9c1447d02aa2371793d4a793c09f37a2c5b08378a75a4f6"}
Oct 29 22:14:31.229568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:31.229165    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2lqth" event={"ID":"7bb945a9-e793-4e53-bb13-38b3eb5022e4","Type":"ContainerStarted","Data":"61e33860f7e1d41d851c85f28caaf0052940a9385b11e72d92ba92bb23188570"}
Oct 29 22:14:31.229568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:31.229258    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:35.628367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.628332    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-2gb5b"]
Oct 29 22:14:35.631549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.631527    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.638249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.638231    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"]
Oct 29 22:14:35.640779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.640764    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:35.646680 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.646663    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-fmpjv\""
Oct 29 22:14:35.647826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.647809    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 22:14:35.647882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.647819    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 22:14:35.657264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.657243    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"]
Oct 29 22:14:35.657651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.657634    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2gb5b"]
Oct 29 22:14:35.659949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.659932    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-g2kdn\""
Oct 29 22:14:35.660036 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.659978    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 29 22:14:35.770107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.770075    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd5bf93c-bbbd-478a-9f00-d115f3d97475-tmp\") pod \"downloads-6dd789c8b7-2gb5b\" (UID: \"cd5bf93c-bbbd-478a-9f00-d115f3d97475\") " pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.770107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.770117    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/19f32a30-891d-46eb-928b-3848afe135d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f9ndh\" (UID: \"19f32a30-891d-46eb-928b-3848afe135d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:35.770340 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.770137    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6llv\" (UniqueName: \"kubernetes.io/projected/cd5bf93c-bbbd-478a-9f00-d115f3d97475-kube-api-access-j6llv\") pod \"downloads-6dd789c8b7-2gb5b\" (UID: \"cd5bf93c-bbbd-478a-9f00-d115f3d97475\") " pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.870953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.870906    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/19f32a30-891d-46eb-928b-3848afe135d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f9ndh\" (UID: \"19f32a30-891d-46eb-928b-3848afe135d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:35.870953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.870956    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6llv\" (UniqueName: \"kubernetes.io/projected/cd5bf93c-bbbd-478a-9f00-d115f3d97475-kube-api-access-j6llv\") pod \"downloads-6dd789c8b7-2gb5b\" (UID: \"cd5bf93c-bbbd-478a-9f00-d115f3d97475\") " pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.871155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.871090    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd5bf93c-bbbd-478a-9f00-d115f3d97475-tmp\") pod \"downloads-6dd789c8b7-2gb5b\" (UID: \"cd5bf93c-bbbd-478a-9f00-d115f3d97475\") " pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.871456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.871437    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd5bf93c-bbbd-478a-9f00-d115f3d97475-tmp\") pod \"downloads-6dd789c8b7-2gb5b\" (UID: \"cd5bf93c-bbbd-478a-9f00-d115f3d97475\") " pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.873283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.873262    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/19f32a30-891d-46eb-928b-3848afe135d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f9ndh\" (UID: \"19f32a30-891d-46eb-928b-3848afe135d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:35.888628 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.888570    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6llv\" (UniqueName: \"kubernetes.io/projected/cd5bf93c-bbbd-478a-9f00-d115f3d97475-kube-api-access-j6llv\") pod \"downloads-6dd789c8b7-2gb5b\" (UID: \"cd5bf93c-bbbd-478a-9f00-d115f3d97475\") " pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.942007 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.941988    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:35.950897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:35.950881    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:36.254492 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.253777    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"]
Oct 29 22:14:36.257334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.257309    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2gb5b"]
Oct 29 22:14:36.266787 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:36.266758    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5bf93c_bbbd_478a_9f00_d115f3d97475.slice/crio-78b3fb1ea5d6dfd5544a3a6ce9d7fdc5785ed75ccd6b23603eb305b815234748 WatchSource:0}: Error finding container 78b3fb1ea5d6dfd5544a3a6ce9d7fdc5785ed75ccd6b23603eb305b815234748: Status 404 returned error can't find the container with id 78b3fb1ea5d6dfd5544a3a6ce9d7fdc5785ed75ccd6b23603eb305b815234748
Oct 29 22:14:36.456961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.456932    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-594947cc9f-q96w2"]
Oct 29 22:14:36.461095 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.461079    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.474396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.474377    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 22:14:36.474500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.474403    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 22:14:36.474500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.474411    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 22:14:36.474500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.474439    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 22:14:36.474610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.474519    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 22:14:36.474610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.474519    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6mtnl\""
Oct 29 22:14:36.477828 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.477808    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-594947cc9f-q96w2"]
Oct 29 22:14:36.577614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577588    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.577782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577623    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.577782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577645    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97bs\" (UniqueName: \"kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.577895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577783    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.577895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577826    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.577895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577853    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.578029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.577935    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679105    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679144    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679168    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97bs\" (UniqueName: \"kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679193    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679220    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679243    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679317    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679744    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.679985 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.679963    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.680070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.680045    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.680110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.680071    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.682023 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.682001    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.682094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.682079    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.695257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.695236    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97bs\" (UniqueName: \"kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs\") pod \"console-594947cc9f-q96w2\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") " pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:36.771975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:36.771701    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:37.070162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:37.070129    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-594947cc9f-q96w2"]
Oct 29 22:14:37.076963 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:37.076435    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3274c90_4ccc_4830_bc73_a1fd787874db.slice/crio-5926cc5eea2a94b17acaf408dfeef0434e1586c761891b7dea7bab16bd2246fd WatchSource:0}: Error finding container 5926cc5eea2a94b17acaf408dfeef0434e1586c761891b7dea7bab16bd2246fd: Status 404 returned error can't find the container with id 5926cc5eea2a94b17acaf408dfeef0434e1586c761891b7dea7bab16bd2246fd
Oct 29 22:14:37.247957 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:37.247899    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594947cc9f-q96w2" event={"ID":"b3274c90-4ccc-4830-bc73-a1fd787874db","Type":"ContainerStarted","Data":"5926cc5eea2a94b17acaf408dfeef0434e1586c761891b7dea7bab16bd2246fd"}
Oct 29 22:14:37.249490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:37.249430    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh" event={"ID":"19f32a30-891d-46eb-928b-3848afe135d4","Type":"ContainerStarted","Data":"428f86e8d73a9698583ff92bf8e6fd37b0b554d4513fbfd1742ae75d3864e675"}
Oct 29 22:14:37.250716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:37.250693    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2gb5b" event={"ID":"cd5bf93c-bbbd-478a-9f00-d115f3d97475","Type":"ContainerStarted","Data":"78b3fb1ea5d6dfd5544a3a6ce9d7fdc5785ed75ccd6b23603eb305b815234748"}
Oct 29 22:14:38.257136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:38.256280    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh" event={"ID":"19f32a30-891d-46eb-928b-3848afe135d4","Type":"ContainerStarted","Data":"8ee759338dc881a8f079aad0b52662a9c33c5d899e192984a7502bac5f4908d6"}
Oct 29 22:14:38.257136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:38.257084    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:38.262782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:38.262760    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh"
Oct 29 22:14:38.330862 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:38.330817    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f9ndh" podStartSLOduration=2.283268091 podStartE2EDuration="3.330805465s" totalImagesPullingTime="1.047537374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:38.298305819 +0000 UTC m=+18.241264525" watchObservedRunningTime="2026-10-29 22:14:38.330805465 +0000 UTC m=+18.273764164"
Oct 29 22:14:39.090439 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.090404    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"]
Oct 29 22:14:39.094034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.094002    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.117544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.117518    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:14:39.118256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.118238    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:14:39.119535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.118314    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 29 22:14:39.119535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.118448    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:14:39.119535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.118524    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4sgzp\""
Oct 29 22:14:39.119535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.119400    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"]
Oct 29 22:14:39.119535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.119472    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 29 22:14:39.198034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.198001    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b29bb251-98b5-49bd-bb0c-bee62b05e783-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.198177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.198046    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b29bb251-98b5-49bd-bb0c-bee62b05e783-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.198177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.198070    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrhj\" (UniqueName: \"kubernetes.io/projected/b29bb251-98b5-49bd-bb0c-bee62b05e783-kube-api-access-lsrhj\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.198177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.198096    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29bb251-98b5-49bd-bb0c-bee62b05e783-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.302060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.301554    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29bb251-98b5-49bd-bb0c-bee62b05e783-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.302060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.301649    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b29bb251-98b5-49bd-bb0c-bee62b05e783-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.302060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.301688    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b29bb251-98b5-49bd-bb0c-bee62b05e783-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.302060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.301709    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrhj\" (UniqueName: \"kubernetes.io/projected/b29bb251-98b5-49bd-bb0c-bee62b05e783-kube-api-access-lsrhj\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.302678 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.302584    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29bb251-98b5-49bd-bb0c-bee62b05e783-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.304581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.304549    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b29bb251-98b5-49bd-bb0c-bee62b05e783-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.304787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.304739    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b29bb251-98b5-49bd-bb0c-bee62b05e783-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.319480 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.319450    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrhj\" (UniqueName: \"kubernetes.io/projected/b29bb251-98b5-49bd-bb0c-bee62b05e783-kube-api-access-lsrhj\") pod \"prometheus-operator-9cd89b74c-r47ld\" (UID: \"b29bb251-98b5-49bd-bb0c-bee62b05e783\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.416509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.416231    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"
Oct 29 22:14:39.724325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.724138    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r47ld"]
Oct 29 22:14:39.730142 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:39.730109    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29bb251_98b5_49bd_bb0c_bee62b05e783.slice/crio-7e62480b2e12ffdbd9c9704805d04ab1e322a8ac3e2281eee425df7842a29fef WatchSource:0}: Error finding container 7e62480b2e12ffdbd9c9704805d04ab1e322a8ac3e2281eee425df7842a29fef: Status 404 returned error can't find the container with id 7e62480b2e12ffdbd9c9704805d04ab1e322a8ac3e2281eee425df7842a29fef
Oct 29 22:14:39.848555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:39.846895    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mh74"
Oct 29 22:14:40.265152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:40.265070    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld" event={"ID":"b29bb251-98b5-49bd-bb0c-bee62b05e783","Type":"ContainerStarted","Data":"7e62480b2e12ffdbd9c9704805d04ab1e322a8ac3e2281eee425df7842a29fef"}
Oct 29 22:14:41.234879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:41.234849    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2lqth"
Oct 29 22:14:41.272295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:41.272096    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594947cc9f-q96w2" event={"ID":"b3274c90-4ccc-4830-bc73-a1fd787874db","Type":"ContainerStarted","Data":"0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99"}
Oct 29 22:14:41.275091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:41.275066    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld" event={"ID":"b29bb251-98b5-49bd-bb0c-bee62b05e783","Type":"ContainerStarted","Data":"ba9dcccc81d49ad123f7f72a0f96191c443d8789d606b6ad60868d493423ab19"}
Oct 29 22:14:41.320556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:41.320500    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-594947cc9f-q96w2" podStartSLOduration=2.022287663 podStartE2EDuration="5.320487682s" totalImagesPullingTime="3.298200019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:41.320253031 +0000 UTC m=+21.263211739" watchObservedRunningTime="2026-10-29 22:14:41.320487682 +0000 UTC m=+21.263446387"
Oct 29 22:14:42.260301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:42.260270    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x4bzm"
Oct 29 22:14:42.284138 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:42.283703    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld" event={"ID":"b29bb251-98b5-49bd-bb0c-bee62b05e783","Type":"ContainerStarted","Data":"a51b9cd57c9094ab53a860dccf43d3204ce9aa2f67e57127e3bebe11e6587d55"}
Oct 29 22:14:42.323140 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:42.323091    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r47ld" podStartSLOduration=1.881872085 podStartE2EDuration="3.323079078s" totalImagesPullingTime="1.441206993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:42.322800971 +0000 UTC m=+22.265759676" watchObservedRunningTime="2026-10-29 22:14:42.323079078 +0000 UTC m=+22.266037782"
Oct 29 22:14:44.634977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.634939    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"]
Oct 29 22:14:44.638409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.638386    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.666877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.666819    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 29 22:14:44.666877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.666862    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 29 22:14:44.667076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.666929    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p24bm\""
Oct 29 22:14:44.667589 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.667562    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"]
Oct 29 22:14:44.687056 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.687028    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qlr9r"]
Oct 29 22:14:44.696825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.696795    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"]
Oct 29 22:14:44.697099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.696965    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.699312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.699293    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.710531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.710509    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"]
Oct 29 22:14:44.713842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.713820    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:14:44.713985 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.713854    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 29 22:14:44.713985 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.713970    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:14:44.714115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.714100    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mlg6x\""
Oct 29 22:14:44.720898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.720877    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 29 22:14:44.721034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.720988    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hdhmn\""
Oct 29 22:14:44.745708 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.745683    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.745825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.745725    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.745876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.745823    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crx2c\" (UniqueName: \"kubernetes.io/projected/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-kube-api-access-crx2c\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.745876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.745862    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.753924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.753893    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:14:44.754018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.753984    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 29 22:14:44.846703 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.846671    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.846703 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.846705    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/151d07f9-8797-4795-94b8-4cee326c14ea-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.846890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.846739    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.846956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.846882    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crx2c\" (UniqueName: \"kubernetes.io/projected/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-kube-api-access-crx2c\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.846956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.846942    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.847040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.846988    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/151d07f9-8797-4795-94b8-4cee326c14ea-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.847040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847016    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-wtmp\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847157 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847139    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.847248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847221    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847269    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.847348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847331    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqg5d\" (UniqueName: \"kubernetes.io/projected/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-kube-api-access-dqg5d\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847387 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847368    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-tls\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847388    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cscdq\" (UniqueName: \"kubernetes.io/projected/151d07f9-8797-4795-94b8-4cee326c14ea-kube-api-access-cscdq\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.847434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847407    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-sys\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847531    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.847629 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847608    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-root\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847653    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.847742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847689    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-textfile\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.847742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.847719    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-metrics-client-ca\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.848279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.848254    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.849899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.849873    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.850080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.850064    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.864298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.864273    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crx2c\" (UniqueName: \"kubernetes.io/projected/b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad-kube-api-access-crx2c\") pod \"openshift-state-metrics-7bd7489957-tnlx5\" (UID: \"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.948523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948447    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-root\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.948523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948501    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-textfile\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.948728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948529    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-metrics-client-ca\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.948728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948555    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.948728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948634    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-root\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.948728 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:44.948668    4451 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Oct 29 22:14:44.948728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948703    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/151d07f9-8797-4795-94b8-4cee326c14ea-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:44.948740    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-tls podName:151d07f9-8797-4795-94b8-4cee326c14ea nodeName:}" failed. No retries permitted until 2026-10-29 22:14:45.448719828 +0000 UTC m=+25.391678516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-tls") pod "kube-state-metrics-5dc8d7d89b-27rcx" (UID: "151d07f9-8797-4795-94b8-4cee326c14ea") : secret "kube-state-metrics-tls" not found
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948784    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948842    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/151d07f9-8797-4795-94b8-4cee326c14ea-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948864    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-wtmp\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948881    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948898    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948939    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948964    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqg5d\" (UniqueName: \"kubernetes.io/projected/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-kube-api-access-dqg5d\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.948983    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-tls\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949002    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cscdq\" (UniqueName: \"kubernetes.io/projected/151d07f9-8797-4795-94b8-4cee326c14ea-kube-api-access-cscdq\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949021    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-sys\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949032    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-textfile\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949076    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-sys\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949223    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-metrics-client-ca\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949425    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/151d07f9-8797-4795-94b8-4cee326c14ea-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949440    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.949747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949721    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.949907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949884    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-wtmp\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.950010 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.949985    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/151d07f9-8797-4795-94b8-4cee326c14ea-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.952024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.951997    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.952460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.952439    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:44.956084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.956061    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"
Oct 29 22:14:44.956700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.956679    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-node-exporter-tls\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.966421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.966399    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqg5d\" (UniqueName: \"kubernetes.io/projected/bd4f7225-ed09-4c35-bd1a-3559ab9a020c-kube-api-access-dqg5d\") pod \"node-exporter-qlr9r\" (UID: \"bd4f7225-ed09-4c35-bd1a-3559ab9a020c\") " pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:44.967387 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:44.967366    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cscdq\" (UniqueName: \"kubernetes.io/projected/151d07f9-8797-4795-94b8-4cee326c14ea-kube-api-access-cscdq\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:45.016017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.015630    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qlr9r"
Oct 29 22:14:45.029672 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:45.029607    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4f7225_ed09_4c35_bd1a_3559ab9a020c.slice/crio-430f92d421d7b175827aa3fb5f16862a80800d35acdcaed9bee1f2216cfe8841 WatchSource:0}: Error finding container 430f92d421d7b175827aa3fb5f16862a80800d35acdcaed9bee1f2216cfe8841: Status 404 returned error can't find the container with id 430f92d421d7b175827aa3fb5f16862a80800d35acdcaed9bee1f2216cfe8841
Oct 29 22:14:45.264529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.264504    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5"]
Oct 29 22:14:45.267077 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:45.267042    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d0efc5_1efd_44b2_a4c7_d5c8f42808ad.slice/crio-d6b0c4a077e43a77f50289f083d8341076308bd0545cc29c04470551aa017391 WatchSource:0}: Error finding container d6b0c4a077e43a77f50289f083d8341076308bd0545cc29c04470551aa017391: Status 404 returned error can't find the container with id d6b0c4a077e43a77f50289f083d8341076308bd0545cc29c04470551aa017391
Oct 29 22:14:45.294984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.294956    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5" event={"ID":"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad","Type":"ContainerStarted","Data":"d6b0c4a077e43a77f50289f083d8341076308bd0545cc29c04470551aa017391"}
Oct 29 22:14:45.297279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.297251    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlr9r" event={"ID":"bd4f7225-ed09-4c35-bd1a-3559ab9a020c","Type":"ContainerStarted","Data":"430f92d421d7b175827aa3fb5f16862a80800d35acdcaed9bee1f2216cfe8841"}
Oct 29 22:14:45.453531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.453494    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:45.456321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.456260    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d07f9-8797-4795-94b8-4cee326c14ea-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-27rcx\" (UID: \"151d07f9-8797-4795-94b8-4cee326c14ea\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:45.623513 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.623457    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"
Oct 29 22:14:45.942478 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.942384    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx"]
Oct 29 22:14:45.946172 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:45.946138    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151d07f9_8797_4795_94b8_4cee326c14ea.slice/crio-45b8dbec96475d15c967e49d98b484fd536d0477d1ad4610cbec7038f5c5a981 WatchSource:0}: Error finding container 45b8dbec96475d15c967e49d98b484fd536d0477d1ad4610cbec7038f5c5a981: Status 404 returned error can't find the container with id 45b8dbec96475d15c967e49d98b484fd536d0477d1ad4610cbec7038f5c5a981
Oct 29 22:14:45.995225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.993504    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:14:45.997895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:45.997398    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.010146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.010125    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 29 22:14:46.010266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.010177    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 29 22:14:46.010266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.010179    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 29 22:14:46.010341 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.010276    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5fq7m\""
Oct 29 22:14:46.019966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.019809    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 29 22:14:46.019966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.019816    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 29 22:14:46.019966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.019889    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 29 22:14:46.019966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.019864    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 29 22:14:46.020552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.020478    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 29 22:14:46.030280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.030257    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 29 22:14:46.031956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.031937    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160168    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160217    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160243    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160269    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160293    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160319    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160341    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160373    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160393    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160408    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160434    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160451    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4gv\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.160712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.160483    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.261656 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261619    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.261815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261669    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.261815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261702    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.261815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261733    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.261815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261758    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.261815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261798    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261825    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4gv\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261851    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261903    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261956    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.261984    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.262012    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.262052    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.262538    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.262868 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.262830    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.263152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.263128    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.264395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.264369    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.266125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.266074    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.266597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.266571    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.267475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.267028    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.267475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.267143    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.267475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.267440    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.268105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.268080    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.269351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.269329    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.283925 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.277816    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.285833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.284522    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4gv\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv\") pod \"alertmanager-main-0\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.303714 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.303689    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5" event={"ID":"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad","Type":"ContainerStarted","Data":"5387c211780d2bb91f83eb0024cad8722b5741aeafe4cc21e27710bec09a1095"}
Oct 29 22:14:46.303828 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.303780    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5" event={"ID":"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad","Type":"ContainerStarted","Data":"2d58c27875e51fc3b72d5a5e20da5b4ed988caa8d924fe63c8e462943afd0d2b"}
Oct 29 22:14:46.305830 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.305804    4451 generic.go:356] "Generic (PLEG): container finished" podID="bd4f7225-ed09-4c35-bd1a-3559ab9a020c" containerID="f86dd98d4bc633068ae4e0bfc693fce607c0a1a68c53a50cb2229fc6b4a7396a" exitCode=0
Oct 29 22:14:46.305960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.305930    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlr9r" event={"ID":"bd4f7225-ed09-4c35-bd1a-3559ab9a020c","Type":"ContainerDied","Data":"f86dd98d4bc633068ae4e0bfc693fce607c0a1a68c53a50cb2229fc6b4a7396a"}
Oct 29 22:14:46.307558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.307527    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx" event={"ID":"151d07f9-8797-4795-94b8-4cee326c14ea","Type":"ContainerStarted","Data":"45b8dbec96475d15c967e49d98b484fd536d0477d1ad4610cbec7038f5c5a981"}
Oct 29 22:14:46.327133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.327110    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:14:46.663689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.663656    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:14:46.667163 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:46.667124    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda373ca33_9005_45c0_9d0b_a04b6dbc036d.slice/crio-bac51200a151047ef75c2c324dcbb66542edf6e718694d35ba330491d0c39f99 WatchSource:0}: Error finding container bac51200a151047ef75c2c324dcbb66542edf6e718694d35ba330491d0c39f99: Status 404 returned error can't find the container with id bac51200a151047ef75c2c324dcbb66542edf6e718694d35ba330491d0c39f99
Oct 29 22:14:46.772445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.772347    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:46.772445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.772413    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:46.779281 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:46.779254    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:47.314948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.314903    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5" event={"ID":"b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad","Type":"ContainerStarted","Data":"9b1d4b1099d8df3bf7030211658c953a3422b0491a2b8db59edc0a2cfa875c4f"}
Oct 29 22:14:47.317378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.317350    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlr9r" event={"ID":"bd4f7225-ed09-4c35-bd1a-3559ab9a020c","Type":"ContainerStarted","Data":"fd7c322caadced0e6aa01e19f813108a239c0073d4976ea1958119f0315aadb6"}
Oct 29 22:14:47.317487 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.317402    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qlr9r" event={"ID":"bd4f7225-ed09-4c35-bd1a-3559ab9a020c","Type":"ContainerStarted","Data":"63b0756664f707a7984f173fa08144477366d68c600ed15244bc8a653bd6fe32"}
Oct 29 22:14:47.319087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.319060    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"bac51200a151047ef75c2c324dcbb66542edf6e718694d35ba330491d0c39f99"}
Oct 29 22:14:47.324110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.324091    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:14:47.392774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.392718    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tnlx5" podStartSLOduration=2.320830078 podStartE2EDuration="3.392700482s" totalImagesPullingTime="1.071870404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:47.351398293 +0000 UTC m=+27.294356998" watchObservedRunningTime="2026-10-29 22:14:47.392700482 +0000 UTC m=+27.335659188"
Oct 29 22:14:47.432518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:47.432041    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qlr9r" podStartSLOduration=2.348737785 podStartE2EDuration="3.4320153s" totalImagesPullingTime="999.857625ms" totalInitContainerRuntime="83.41989ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:47.431638691 +0000 UTC m=+27.374597397" watchObservedRunningTime="2026-10-29 22:14:47.4320153 +0000 UTC m=+27.374974011"
Oct 29 22:14:48.330597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:48.330554    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx" event={"ID":"151d07f9-8797-4795-94b8-4cee326c14ea","Type":"ContainerStarted","Data":"f8ab3ad383c2ab8bdec4f72856b0ca8685fd11ed7cb9b965707b6aaa5c9b6d14"}
Oct 29 22:14:48.331090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:48.330659    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx" event={"ID":"151d07f9-8797-4795-94b8-4cee326c14ea","Type":"ContainerStarted","Data":"e4ecb18f3e9ece0e2e9c5da63963dbfa5ae100b80239a6e22043a0bc2bed8a00"}
Oct 29 22:14:48.331090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:48.330683    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx" event={"ID":"151d07f9-8797-4795-94b8-4cee326c14ea","Type":"ContainerStarted","Data":"1983d9a9c86c62df1fdbf31f27553c93315d0a6062cf8ba68d120a32192caf47"}
Oct 29 22:14:48.332260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:48.332233    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="6feb03babe17187d4475e7e4d55eaaac0c825c1fe3955b7fea3a82ee721d9895" exitCode=0
Oct 29 22:14:48.332370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:48.332330    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"6feb03babe17187d4475e7e4d55eaaac0c825c1fe3955b7fea3a82ee721d9895"}
Oct 29 22:14:48.367011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:48.366882    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-27rcx" podStartSLOduration=3.038290553 podStartE2EDuration="4.366866612s" totalImagesPullingTime="1.328576059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:48.366117387 +0000 UTC m=+28.309076106" watchObservedRunningTime="2026-10-29 22:14:48.366866612 +0000 UTC m=+28.309825319"
Oct 29 22:14:48.742321 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:14:48.742234    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:14:49.497373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.497337    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"]
Oct 29 22:14:49.513286 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.512837    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.516505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.516480    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"]
Oct 29 22:14:49.537466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.537439    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 29 22:14:49.537466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.537448    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mdtgl\""
Oct 29 22:14:49.692647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.692608    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62n46\" (UniqueName: \"kubernetes.io/projected/7b25a6f4-8818-4df3-84d1-0302f82de501-kube-api-access-62n46\") pod \"monitoring-plugin-77f4f76594-76vbk\" (UID: \"7b25a6f4-8818-4df3-84d1-0302f82de501\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.692824 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.692800    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b25a6f4-8818-4df3-84d1-0302f82de501-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-76vbk\" (UID: \"7b25a6f4-8818-4df3-84d1-0302f82de501\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.793423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.793341    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62n46\" (UniqueName: \"kubernetes.io/projected/7b25a6f4-8818-4df3-84d1-0302f82de501-kube-api-access-62n46\") pod \"monitoring-plugin-77f4f76594-76vbk\" (UID: \"7b25a6f4-8818-4df3-84d1-0302f82de501\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.793423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.793395    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b25a6f4-8818-4df3-84d1-0302f82de501-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-76vbk\" (UID: \"7b25a6f4-8818-4df3-84d1-0302f82de501\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.795853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.795822    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b25a6f4-8818-4df3-84d1-0302f82de501-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-76vbk\" (UID: \"7b25a6f4-8818-4df3-84d1-0302f82de501\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.812049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.812021    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62n46\" (UniqueName: \"kubernetes.io/projected/7b25a6f4-8818-4df3-84d1-0302f82de501-kube-api-access-62n46\") pod \"monitoring-plugin-77f4f76594-76vbk\" (UID: \"7b25a6f4-8818-4df3-84d1-0302f82de501\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.841782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.841752    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:49.959517 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.959488    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"]
Oct 29 22:14:49.972962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.972938    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.977537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.977511    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"]
Oct 29 22:14:49.985889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.985816    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 29 22:14:49.985889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.985868    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 29 22:14:49.986038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.985946    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 29 22:14:49.986038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.985948    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 29 22:14:49.995783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.995753    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-telemeter-client-tls\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.995783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.995771    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-f8lwp\""
Oct 29 22:14:49.995945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.995830    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-metrics-client-ca\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.995983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.995948    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-secret-telemeter-client\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.996021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.995978    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qr2\" (UniqueName: \"kubernetes.io/projected/67122eae-b9ce-43a7-b0de-691a8ee97a21-kube-api-access-w6qr2\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.996021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.996007    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-serving-certs-ca-bundle\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.996079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.996030    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.996079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.996052    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-federate-client-tls\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.996159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.996123    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:49.996385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:49.996357    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 29 22:14:50.006125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.006107    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 29 22:14:50.096531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096501    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-telemeter-client-tls\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.096709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096553    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-metrics-client-ca\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.096709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096586    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-secret-telemeter-client\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.096709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096612    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qr2\" (UniqueName: \"kubernetes.io/projected/67122eae-b9ce-43a7-b0de-691a8ee97a21-kube-api-access-w6qr2\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.096709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096640    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-serving-certs-ca-bundle\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.096709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096662    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.096709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096692    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-federate-client-tls\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.097436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.096732    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.097573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.097549    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.098076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.098045    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-serving-certs-ca-bundle\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.098373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.098351    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67122eae-b9ce-43a7-b0de-691a8ee97a21-metrics-client-ca\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.099648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.099595    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-telemeter-client-tls\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.099786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.099762    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.099870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.099854    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-secret-telemeter-client\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.100053 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.100036    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67122eae-b9ce-43a7-b0de-691a8ee97a21-federate-client-tls\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.114316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.114294    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qr2\" (UniqueName: \"kubernetes.io/projected/67122eae-b9ce-43a7-b0de-691a8ee97a21-kube-api-access-w6qr2\") pod \"telemeter-client-5cd9b7f765-g5nwx\" (UID: \"67122eae-b9ce-43a7-b0de-691a8ee97a21\") " pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.155829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.155797    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"]
Oct 29 22:14:50.192014 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:50.191980    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b25a6f4_8818_4df3_84d1_0302f82de501.slice/crio-fee6e75d5e485201434ac5607043236f1f101a7869311a1aec8abae17bc06edd WatchSource:0}: Error finding container fee6e75d5e485201434ac5607043236f1f101a7869311a1aec8abae17bc06edd: Status 404 returned error can't find the container with id fee6e75d5e485201434ac5607043236f1f101a7869311a1aec8abae17bc06edd
Oct 29 22:14:50.333432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.333396    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"
Oct 29 22:14:50.343260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.342744    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"a0a60a93d6f91cb98d701692305fd68817b66dc3c231a1087957dc0cb8174b13"}
Oct 29 22:14:50.344306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.344277    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk" event={"ID":"7b25a6f4-8818-4df3-84d1-0302f82de501","Type":"ContainerStarted","Data":"fee6e75d5e485201434ac5607043236f1f101a7869311a1aec8abae17bc06edd"}
Oct 29 22:14:50.676773 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:50.676747    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx"]
Oct 29 22:14:50.680994 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:50.680963    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67122eae_b9ce_43a7_b0de_691a8ee97a21.slice/crio-d7ecd3682084f98cb06d1de97208081a9c714d3ad8f2c566e1f95a4df0531d20 WatchSource:0}: Error finding container d7ecd3682084f98cb06d1de97208081a9c714d3ad8f2c566e1f95a4df0531d20: Status 404 returned error can't find the container with id d7ecd3682084f98cb06d1de97208081a9c714d3ad8f2c566e1f95a4df0531d20
Oct 29 22:14:51.353878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.353829    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"40355d60af29031d27cba5e396e7473d51e282a400b54bf3a744a2862a9e7a3e"}
Oct 29 22:14:51.354088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.353930    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"6dc1434d31867cda062ce4dee3dbf313f12b4ec6f8adb594fdf471e8a7183f14"}
Oct 29 22:14:51.354088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.353973    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"1db8e1158c9eaa4a43a577c4e7f94516d015a303c6261092dd9553a1f0ab80f9"}
Oct 29 22:14:51.354088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.354014    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"5a53b9d49efa9078b0f157b0d739836fdb40e95849316b975f93456f4596bb66"}
Oct 29 22:14:51.355288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.355244    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx" event={"ID":"67122eae-b9ce-43a7-b0de-691a8ee97a21","Type":"ContainerStarted","Data":"d7ecd3682084f98cb06d1de97208081a9c714d3ad8f2c566e1f95a4df0531d20"}
Oct 29 22:14:51.543606 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.543568    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:14:51.561813 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.561776    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.574298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.574272    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:14:51.579042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.579021    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 22:14:51.579173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.579063    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 22:14:51.579173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.579104    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 22:14:51.579173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.579126    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 22:14:51.579325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.579207    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 22:14:51.581194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.581177    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 22:14:51.581194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.581186    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 22:14:51.581348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.581256    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ecd5mvg2qg51g\""
Oct 29 22:14:51.581348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.581321    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7xf7\""
Oct 29 22:14:51.607760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607689    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607729    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607758    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607783    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607812    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607839    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607863    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607897    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607941    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.607994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.607979    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608005    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608029    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608052    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608083    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608114    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608154    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608196    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6k5\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608233    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608279    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.608415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.608304    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.683768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.683711    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 22:14:51.684302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.684000    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 22:14:51.684787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.684760    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 22:14:51.684932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.684894    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 22:14:51.684999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.684967    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 22:14:51.685073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.685036    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 22:14:51.695776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.695747    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 22:14:51.709426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.709394    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710606 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710558    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710622    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710661    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710685    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710703    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710720    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710749    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710778    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710807    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710845    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.710949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710903    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.710977    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711008    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711034    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711060    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711091    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711124    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711165    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711207    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6k5\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.711257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.711235    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.714405    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.714649    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.714763    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.714882    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.715231    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.715426    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.715701    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.717964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.716339    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.718890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.718689    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.719160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.719122    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.719903 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.719827    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.722088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.722066    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.725420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.725396    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.725611 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.725589    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.732545 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.732498    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6k5\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.738675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.738152    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.738675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.738400    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.738675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.738638    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.739818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.739772    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.742076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.742055    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:51.997739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:51.997661    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:14:52.361848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:52.361812    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:14:52.445229 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:14:52.445181    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7666d69_65dc_42f3_8702_ffb3d664f29b.slice/crio-8937da17a284926dc48cc26a4ce068a2c768f6fbe83c35c040d5d4088a159723 WatchSource:0}: Error finding container 8937da17a284926dc48cc26a4ce068a2c768f6fbe83c35c040d5d4088a159723: Status 404 returned error can't find the container with id 8937da17a284926dc48cc26a4ce068a2c768f6fbe83c35c040d5d4088a159723
Oct 29 22:14:53.368785 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.368745    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerStarted","Data":"d909bf7f0173a2a6cbcf2f5a64f154dcfd11146ca13176478c54f4c112fe908f"}
Oct 29 22:14:53.370416 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.370389    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk" event={"ID":"7b25a6f4-8818-4df3-84d1-0302f82de501","Type":"ContainerStarted","Data":"1adc9609ce0a3d262f57c332218e4b7a06e1c7af13800b9621c24b588f51f500"}
Oct 29 22:14:53.370558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.370538    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:53.372230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.372200    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05" exitCode=0
Oct 29 22:14:53.372312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.372242    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"}
Oct 29 22:14:53.372312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.372285    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"8937da17a284926dc48cc26a4ce068a2c768f6fbe83c35c040d5d4088a159723"}
Oct 29 22:14:53.376722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.376703    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk"
Oct 29 22:14:53.460446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.460394    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-76vbk" podStartSLOduration=2.173203018 podStartE2EDuration="4.460377502s" totalImagesPullingTime="2.287174484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:53.458156748 +0000 UTC m=+33.401115454" watchObservedRunningTime="2026-10-29 22:14:53.460377502 +0000 UTC m=+33.403336201"
Oct 29 22:14:53.461779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:53.461733    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.680194518 podStartE2EDuration="8.461715016s" totalImagesPullingTime="4.758696655s" totalInitContainerRuntime="22.823843ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:53.423542853 +0000 UTC m=+33.366501558" watchObservedRunningTime="2026-10-29 22:14:53.461715016 +0000 UTC m=+33.404673721"
Oct 29 22:14:54.379899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:54.379853    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx" event={"ID":"67122eae-b9ce-43a7-b0de-691a8ee97a21","Type":"ContainerStarted","Data":"74c02f0223d0b9d04848279b03834f8d5f1625c89351ea25af3688b545c1ab9f"}
Oct 29 22:14:54.380390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:54.379942    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx" event={"ID":"67122eae-b9ce-43a7-b0de-691a8ee97a21","Type":"ContainerStarted","Data":"7abd5e9d2deaa0253b90ee82c3f1e72b2c02dc434d896fa2073164ef86079ae0"}
Oct 29 22:14:54.380390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:54.379970    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx" event={"ID":"67122eae-b9ce-43a7-b0de-691a8ee97a21","Type":"ContainerStarted","Data":"a7b25097449da1734b11c135037eee33cc95da43e6707e14ef17669f8758cc89"}
Oct 29 22:14:54.437886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:54.437817    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5cd9b7f765-g5nwx" podStartSLOduration=2.73231576 podStartE2EDuration="5.437799429s" totalImagesPullingTime="2.705483669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:54.426630208 +0000 UTC m=+34.369588926" watchObservedRunningTime="2026-10-29 22:14:54.437799429 +0000 UTC m=+34.380758135"
Oct 29 22:14:55.388159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:55.388123    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qp2mt"
Oct 29 22:14:55.390652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:55.390574    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2gb5b" event={"ID":"cd5bf93c-bbbd-478a-9f00-d115f3d97475","Type":"ContainerStarted","Data":"78eabf2966aa6dcba2237597afac6d5dafef213c5b6c996db38bb6ef345ef7fe"}
Oct 29 22:14:55.390652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:55.390627    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:55.406311 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:55.406286    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-2gb5b"
Oct 29 22:14:58.417778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:58.417643    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"}
Oct 29 22:14:58.417778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:14:58.417727    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"}
Oct 29 22:15:00.449818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.449749    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-2gb5b" podStartSLOduration=7.229616163 podStartE2EDuration="25.449730791s" totalImagesPullingTime="18.220114628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:55.456493806 +0000 UTC m=+35.399452511" watchObservedRunningTime="2026-10-29 22:15:00.449730791 +0000 UTC m=+40.392689499"
Oct 29 22:15:00.451244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.451219    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bfd86cd7c-9wsgc"]
Oct 29 22:15:00.477264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.477241    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfd86cd7c-9wsgc"]
Oct 29 22:15:00.477403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.477353    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.498706 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.498676    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.498840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.498725    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.498887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.498846    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.498956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.498883    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.499062 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.499040    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.499123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.499078    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpg9s\" (UniqueName: \"kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.499181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.499120    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.499181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.499164    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.519834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.519803    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 22:15:00.600646 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600620    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.600808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600659    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpg9s\" (UniqueName: \"kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.600808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600693    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.600808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600729    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.600808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600780    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.600808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600809    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.601102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600844    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.601102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.600872    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.601770 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.601626    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.603170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.601872    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.603170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.602190    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.603170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.602577    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.603170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.603053    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.603698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.603674    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.604857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.604833    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.618956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.618934    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpg9s\" (UniqueName: \"kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s\") pod \"console-7bfd86cd7c-9wsgc\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") " pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:00.812570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:00.812536    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:01.138049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.137301    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfd86cd7c-9wsgc"]
Oct 29 22:15:01.145017 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:15:01.144972    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf613a96d_ae8d_448b_b772_b417eb6aae71.slice/crio-3249282c3cd0a18a26a0582cbf87c0517b6953b928cbd751b125d89a5093bd89 WatchSource:0}: Error finding container 3249282c3cd0a18a26a0582cbf87c0517b6953b928cbd751b125d89a5093bd89: Status 404 returned error can't find the container with id 3249282c3cd0a18a26a0582cbf87c0517b6953b928cbd751b125d89a5093bd89
Oct 29 22:15:01.438072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.438034    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"}
Oct 29 22:15:01.438260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.438121    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"}
Oct 29 22:15:01.438260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.438165    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"}
Oct 29 22:15:01.438260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.438202    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerStarted","Data":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"}
Oct 29 22:15:01.439904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.439873    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfd86cd7c-9wsgc" event={"ID":"f613a96d-ae8d-448b-b772-b417eb6aae71","Type":"ContainerStarted","Data":"4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd"}
Oct 29 22:15:01.440050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.439952    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfd86cd7c-9wsgc" event={"ID":"f613a96d-ae8d-448b-b772-b417eb6aae71","Type":"ContainerStarted","Data":"3249282c3cd0a18a26a0582cbf87c0517b6953b928cbd751b125d89a5093bd89"}
Oct 29 22:15:01.483720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.483660    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.516537205 podStartE2EDuration="10.483640158s" totalImagesPullingTime="6.93995167s" totalInitContainerRuntime="27.151283ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:01.481585203 +0000 UTC m=+41.424543910" watchObservedRunningTime="2026-10-29 22:15:01.483640158 +0000 UTC m=+41.426598866"
Oct 29 22:15:01.518312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.518259    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7bfd86cd7c-9wsgc" podStartSLOduration=1.518244707 podStartE2EDuration="1.518244707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:01.516650324 +0000 UTC m=+41.459609039" watchObservedRunningTime="2026-10-29 22:15:01.518244707 +0000 UTC m=+41.461203411"
Oct 29 22:15:01.998684 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:01.998650    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:10.813624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:10.813535    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:10.813624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:10.813587    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:10.818406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:10.818382    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:11.481485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:11.481457    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:15:11.544295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:11.544256    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-594947cc9f-q96w2"]
Oct 29 22:15:18.414145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:18.414112    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:18.461274 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:18.461251    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:18.518675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:18.518645    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:18.738938 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:18.738841    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:15:35.338728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.338694    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:15:35.339171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.338989    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="alertmanager" containerID="cri-o://a0a60a93d6f91cb98d701692305fd68817b66dc3c231a1087957dc0cb8174b13" gracePeriod=120
Oct 29 22:15:35.339171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.339026    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy" containerID="cri-o://6dc1434d31867cda062ce4dee3dbf313f12b4ec6f8adb594fdf471e8a7183f14" gracePeriod=120
Oct 29 22:15:35.339171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.339041    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-web" containerID="cri-o://1db8e1158c9eaa4a43a577c4e7f94516d015a303c6261092dd9553a1f0ab80f9" gracePeriod=120
Oct 29 22:15:35.339171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.339077    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="config-reloader" containerID="cri-o://5a53b9d49efa9078b0f157b0d739836fdb40e95849316b975f93456f4596bb66" gracePeriod=120
Oct 29 22:15:35.339171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.339040    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="prom-label-proxy" containerID="cri-o://d909bf7f0173a2a6cbcf2f5a64f154dcfd11146ca13176478c54f4c112fe908f" gracePeriod=120
Oct 29 22:15:35.339171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.339083    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-metric" containerID="cri-o://40355d60af29031d27cba5e396e7473d51e282a400b54bf3a744a2862a9e7a3e" gracePeriod=120
Oct 29 22:15:35.576538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576505    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="d909bf7f0173a2a6cbcf2f5a64f154dcfd11146ca13176478c54f4c112fe908f" exitCode=0
Oct 29 22:15:35.576538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576530    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="40355d60af29031d27cba5e396e7473d51e282a400b54bf3a744a2862a9e7a3e" exitCode=0
Oct 29 22:15:35.576538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576537    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="6dc1434d31867cda062ce4dee3dbf313f12b4ec6f8adb594fdf471e8a7183f14" exitCode=0
Oct 29 22:15:35.576538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576542    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="5a53b9d49efa9078b0f157b0d739836fdb40e95849316b975f93456f4596bb66" exitCode=0
Oct 29 22:15:35.576772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576547    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="a0a60a93d6f91cb98d701692305fd68817b66dc3c231a1087957dc0cb8174b13" exitCode=0
Oct 29 22:15:35.576772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576574    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"d909bf7f0173a2a6cbcf2f5a64f154dcfd11146ca13176478c54f4c112fe908f"}
Oct 29 22:15:35.576772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576637    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"40355d60af29031d27cba5e396e7473d51e282a400b54bf3a744a2862a9e7a3e"}
Oct 29 22:15:35.576772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576662    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"6dc1434d31867cda062ce4dee3dbf313f12b4ec6f8adb594fdf471e8a7183f14"}
Oct 29 22:15:35.576772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576701    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"5a53b9d49efa9078b0f157b0d739836fdb40e95849316b975f93456f4596bb66"}
Oct 29 22:15:35.576772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:35.576725    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"a0a60a93d6f91cb98d701692305fd68817b66dc3c231a1087957dc0cb8174b13"}
Oct 29 22:15:36.564230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.564195    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-594947cc9f-q96w2" podUID="b3274c90-4ccc-4830-bc73-a1fd787874db" containerName="console" containerID="cri-o://0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99" gracePeriod=15
Oct 29 22:15:36.583447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.583425    4451 generic.go:356] "Generic (PLEG): container finished" podID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerID="1db8e1158c9eaa4a43a577c4e7f94516d015a303c6261092dd9553a1f0ab80f9" exitCode=0
Oct 29 22:15:36.583539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.583490    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"1db8e1158c9eaa4a43a577c4e7f94516d015a303c6261092dd9553a1f0ab80f9"}
Oct 29 22:15:36.583578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.583548    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a373ca33-9005-45c0-9d0b-a04b6dbc036d","Type":"ContainerDied","Data":"bac51200a151047ef75c2c324dcbb66542edf6e718694d35ba330491d0c39f99"}
Oct 29 22:15:36.583578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.583573    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac51200a151047ef75c2c324dcbb66542edf6e718694d35ba330491d0c39f99"
Oct 29 22:15:36.593683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.593666    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:36.715790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.715746    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.715804    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.715851    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.715890    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.715940    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.715986    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716011    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716050    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716105    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716140    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716187    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716234    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716267    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv\") pod \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\" (UID: \"a373ca33-9005-45c0-9d0b-a04b6dbc036d\") "
Oct 29 22:15:36.716613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.716579    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:36.717661 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.717630    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:36.717861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.717817    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:36.720591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.720529    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:36.720722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.720684    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.721016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.720822    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:36.721155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.720956    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.721438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.721413    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.721694 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.721652    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "kube-api-access-dp4gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:36.722141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.722114    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.723880 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.723773    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.726376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.726348    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.733273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.733250    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config" pod "a373ca33-9005-45c0-9d0b-a04b6dbc036d" (UID: "a373ca33-9005-45c0-9d0b-a04b6dbc036d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:36.817310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817284    4451 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817308    4451 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-cluster-tls-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817320    4451 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-web-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817330    4451 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-main-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817339    4451 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817347    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-metrics-client-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817356    4451 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-tls-assets\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817364    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4gv\" (UniqueName: \"kubernetes.io/projected/a373ca33-9005-45c0-9d0b-a04b6dbc036d-kube-api-access-dp4gv\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817373    4451 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817381    4451 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-out\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817390    4451 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817398    4451 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a373ca33-9005-45c0-9d0b-a04b6dbc036d-config-volume\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.817465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.817405    4451 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a373ca33-9005-45c0-9d0b-a04b6dbc036d-alertmanager-main-db\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:36.842846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:36.842814    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:15:37.018621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018541    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.018621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018609    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs\" (UniqueName: \"kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.018834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018716    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.018834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018757    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.018834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018807    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.019028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018834    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.019028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.018868    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp\") pod \"b3274c90-4ccc-4830-bc73-a1fd787874db\" (UID: \"b3274c90-4ccc-4830-bc73-a1fd787874db\") "
Oct 29 22:15:37.019142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019009    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:37.019198 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019179    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:37.019250 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019218    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:37.019305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019280    4451 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-oauth-serving-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.019305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019294    4451 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-console-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.019399 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019308    4451 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3274c90-4ccc-4830-bc73-a1fd787874db-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.019453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.019396    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:37.020854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.020833    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:37.020966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.020875    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "kube-api-access-n97bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:37.020966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.020929    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert" pod "b3274c90-4ccc-4830-bc73-a1fd787874db" (UID: "b3274c90-4ccc-4830-bc73-a1fd787874db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:37.119689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.119656    4451 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-serving-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.119689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.119684    4451 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3274c90-4ccc-4830-bc73-a1fd787874db-tmp\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.119865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.119700    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n97bs\" (UniqueName: \"kubernetes.io/projected/b3274c90-4ccc-4830-bc73-a1fd787874db-kube-api-access-n97bs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.119865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.119713    4451 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3274c90-4ccc-4830-bc73-a1fd787874db-console-oauth-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:37.588691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.588657    4451 generic.go:356] "Generic (PLEG): container finished" podID="b3274c90-4ccc-4830-bc73-a1fd787874db" containerID="0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99" exitCode=0
Oct 29 22:15:37.589090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.588701    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594947cc9f-q96w2"
Oct 29 22:15:37.589090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.588746    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594947cc9f-q96w2" event={"ID":"b3274c90-4ccc-4830-bc73-a1fd787874db","Type":"ContainerDied","Data":"0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99"}
Oct 29 22:15:37.589090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.588795    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594947cc9f-q96w2" event={"ID":"b3274c90-4ccc-4830-bc73-a1fd787874db","Type":"ContainerDied","Data":"5926cc5eea2a94b17acaf408dfeef0434e1586c761891b7dea7bab16bd2246fd"}
Oct 29 22:15:37.589090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.588827    4451 scope.go:122] "RemoveContainer" containerID="0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99"
Oct 29 22:15:37.589090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.589028    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.597161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.597143    4451 scope.go:122] "RemoveContainer" containerID="0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99"
Oct 29 22:15:37.597475 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:37.597440    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99\": container with ID starting with 0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99 not found: ID does not exist" containerID="0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99"
Oct 29 22:15:37.597586 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.597483    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99"} err="failed to get container status \"0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99\": rpc error: code = NotFound desc = could not find container \"0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99\": container with ID starting with 0d773ddcefe1f99ec4ba27892367681ea72a829a284d0d1cb0b9f00842b88f99 not found: ID does not exist"
Oct 29 22:15:37.644011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.643986    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:15:37.654290 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.654259    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:15:37.682097 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.682039    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-594947cc9f-q96w2"]
Oct 29 22:15:37.685873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:37.685854    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-594947cc9f-q96w2"]
Oct 29 22:15:38.160548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:38.160517    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" path="/var/lib/kubelet/pods/a373ca33-9005-45c0-9d0b-a04b6dbc036d/volumes"
Oct 29 22:15:38.161198 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:38.161182    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3274c90-4ccc-4830-bc73-a1fd787874db" path="/var/lib/kubelet/pods/b3274c90-4ccc-4830-bc73-a1fd787874db/volumes"
Oct 29 22:15:39.922334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.922294    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:15:39.923112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.922685    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="prometheus" containerID="cri-o://b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476" gracePeriod=600
Oct 29 22:15:39.923112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.922866    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="thanos-sidecar" containerID="cri-o://64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa" gracePeriod=600
Oct 29 22:15:39.923112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.923012    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-web" containerID="cri-o://c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999" gracePeriod=600
Oct 29 22:15:39.923112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.923076    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552" gracePeriod=600
Oct 29 22:15:39.923396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.922846    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="config-reloader" containerID="cri-o://0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1" gracePeriod=600
Oct 29 22:15:39.924330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:39.924299    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy" containerID="cri-o://c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b" gracePeriod=600
Oct 29 22:15:40.203390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.203322    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.246031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.245982    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246025    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246046    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246082    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246109    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246138    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246174    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246202    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246250    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246278    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246322    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246362    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246390    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246401    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246432    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246461    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246492    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246526    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246549    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246558    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246585    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246620    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out\") pod \"b7666d69-65dc-42f3-8702-ffb3d664f29b\" (UID: \"b7666d69-65dc-42f3-8702-ffb3d664f29b\") "
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246642    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246853    4451 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246872    4451 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.246885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.246888    4451 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.247696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.247003    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.247848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.247824    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.248987 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.248448    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.249087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.248979    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.249087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.249064    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:40.249434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.249406    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:40.249755 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.249706    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "kube-api-access-kw6k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:40.251447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.251417    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.251653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.251624    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.252193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.252173    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:40.252193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.252181    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.252357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.252213    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.252357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.252226    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.252357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.252230    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:40.252357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.252272    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.253695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.253663    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.263666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.263638    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config" pod "b7666d69-65dc-42f3-8702-ffb3d664f29b" (UID: "b7666d69-65dc-42f3-8702-ffb3d664f29b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:40.347766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347736    4451 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-web-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.347766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347760    4451 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.347766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347772    4451 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347781    4451 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347791    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw6k5\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-kube-api-access-kw6k5\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347800    4451 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347810    4451 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-kube-rbac-proxy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347818    4451 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347827    4451 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347835    4451 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7666d69-65dc-42f3-8702-ffb3d664f29b-tls-assets\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347843    4451 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347851    4451 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-config-out\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347860    4451 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-grpc-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347869    4451 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7666d69-65dc-42f3-8702-ffb3d664f29b-prometheus-k8s-db\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347876    4451 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-secret-metrics-client-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347884    4451 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7666d69-65dc-42f3-8702-ffb3d664f29b-configmap-metrics-client-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.348071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.347892    4451 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7666d69-65dc-42f3-8702-ffb3d664f29b-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:15:40.606316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606279    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552" exitCode=0
Oct 29 22:15:40.606316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606312    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b" exitCode=0
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606320    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999" exitCode=0
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606331    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa" exitCode=0
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606339    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1" exitCode=0
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606346    4451 generic.go:356] "Generic (PLEG): container finished" podID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476" exitCode=0
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606367    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"}
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606455    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"}
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606379    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606500    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"}
Oct 29 22:15:40.606544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606539    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.606890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606542    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"}
Oct 29 22:15:40.606890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606668    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"}
Oct 29 22:15:40.606890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606715    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"}
Oct 29 22:15:40.606890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.606749    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7666d69-65dc-42f3-8702-ffb3d664f29b","Type":"ContainerDied","Data":"8937da17a284926dc48cc26a4ce068a2c768f6fbe83c35c040d5d4088a159723"}
Oct 29 22:15:40.614728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.614590    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.621424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.621407    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.627777 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.627761    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.633940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.633904    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.642238 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.642222    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.649039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.649018    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.653742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.653723    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:15:40.657436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.657420    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.657689 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.657672    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.657737 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.657698    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"} err="failed to get container status \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist"
Oct 29 22:15:40.657737 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.657713    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.657974 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.657950    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.658041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.657980    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"} err="failed to get container status \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist"
Oct 29 22:15:40.658041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658000    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.658266 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.658249    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.658301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658272    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"} err="failed to get container status \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist"
Oct 29 22:15:40.658301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658287    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.658512 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.658498    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.658550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658515    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"} err="failed to get container status \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist"
Oct 29 22:15:40.658550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658529    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.658722 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.658704    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.658761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658726    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"} err="failed to get container status \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist"
Oct 29 22:15:40.658761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658739    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.658962 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.658942    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.659034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658971    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"} err="failed to get container status \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist"
Oct 29 22:15:40.659034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.658992    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.659256 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:40.659239    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.659294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659263    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"} err="failed to get container status \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist"
Oct 29 22:15:40.659294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659278    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.659492 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659477    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"} err="failed to get container status \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist"
Oct 29 22:15:40.659543 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659493    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.659688 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659671    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"} err="failed to get container status \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist"
Oct 29 22:15:40.659738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659688    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.659894 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659876    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"} err="failed to get container status \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist"
Oct 29 22:15:40.659998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.659896    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.660131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660114    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"} err="failed to get container status \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist"
Oct 29 22:15:40.660177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660130    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.660321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660303    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"} err="failed to get container status \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist"
Oct 29 22:15:40.660363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660320    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.660515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660500    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"} err="failed to get container status \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist"
Oct 29 22:15:40.660556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660515    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.660704 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660686    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"} err="failed to get container status \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist"
Oct 29 22:15:40.660742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660704    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.660873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660858    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"} err="failed to get container status \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist"
Oct 29 22:15:40.660935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.660872    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.661070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661054    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"} err="failed to get container status \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist"
Oct 29 22:15:40.661114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661070    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.661254 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661239    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"} err="failed to get container status \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist"
Oct 29 22:15:40.661297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661254    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.661453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661439    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"} err="failed to get container status \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist"
Oct 29 22:15:40.661498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661453    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.661638 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661618    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"} err="failed to get container status \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist"
Oct 29 22:15:40.661709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661639    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.661846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661829    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"} err="failed to get container status \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist"
Oct 29 22:15:40.661963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.661848    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.662094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662068    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"} err="failed to get container status \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist"
Oct 29 22:15:40.662094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662087    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.662352 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662326    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"} err="failed to get container status \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist"
Oct 29 22:15:40.662457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662353    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.662627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662604    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"} err="failed to get container status \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist"
Oct 29 22:15:40.662674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662630    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.662963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662896    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"} err="failed to get container status \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist"
Oct 29 22:15:40.663106 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.662974    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.663291 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663270    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"} err="failed to get container status \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist"
Oct 29 22:15:40.663350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663291    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.663546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663514    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"} err="failed to get container status \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist"
Oct 29 22:15:40.663546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663536    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:15:40.663546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663543    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.663796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663780    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"} err="failed to get container status \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist"
Oct 29 22:15:40.663847 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.663796    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.664071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664054    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"} err="failed to get container status \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist"
Oct 29 22:15:40.664071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664070    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.664266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664250    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"} err="failed to get container status \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist"
Oct 29 22:15:40.664316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664266    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.664468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664452    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"} err="failed to get container status \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist"
Oct 29 22:15:40.664511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664468    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.664689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664672    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"} err="failed to get container status \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist"
Oct 29 22:15:40.664689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664688    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.664872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664857    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"} err="failed to get container status \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist"
Oct 29 22:15:40.664929 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.664872    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.665073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665057    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"} err="failed to get container status \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist"
Oct 29 22:15:40.665115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665073    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.665244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665229    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"} err="failed to get container status \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist"
Oct 29 22:15:40.665288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665244    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.665470 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665452    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"} err="failed to get container status \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist"
Oct 29 22:15:40.665535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665470    4451 scope.go:122] "RemoveContainer" containerID="b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"
Oct 29 22:15:40.665689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665670    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552"} err="failed to get container status \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": rpc error: code = NotFound desc = could not find container \"b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552\": container with ID starting with b41935cf176b4d5562d9a8c7e608eada8236de984c9ee794d468643a48b4f552 not found: ID does not exist"
Oct 29 22:15:40.665689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665688    4451 scope.go:122] "RemoveContainer" containerID="c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"
Oct 29 22:15:40.665833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665818    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b"} err="failed to get container status \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": rpc error: code = NotFound desc = could not find container \"c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b\": container with ID starting with c5232f15d3d4cb17ca8bf7d0d2d3a886e4475edf591ae241ff15015dcb1b111b not found: ID does not exist"
Oct 29 22:15:40.665873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.665833    4451 scope.go:122] "RemoveContainer" containerID="c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"
Oct 29 22:15:40.666061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666045    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999"} err="failed to get container status \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": rpc error: code = NotFound desc = could not find container \"c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999\": container with ID starting with c5fd8b8cc963809a8cb740c95d1893ea37161734941a1205940b229d661a6999 not found: ID does not exist"
Oct 29 22:15:40.666061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666060    4451 scope.go:122] "RemoveContainer" containerID="64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"
Oct 29 22:15:40.666230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666215    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa"} err="failed to get container status \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": rpc error: code = NotFound desc = could not find container \"64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa\": container with ID starting with 64edbf3e8e07757816850610e21b691310ebddc40e388bfa13704ac7d1b1dbfa not found: ID does not exist"
Oct 29 22:15:40.666230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666229    4451 scope.go:122] "RemoveContainer" containerID="0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"
Oct 29 22:15:40.666405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666389    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1"} err="failed to get container status \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": rpc error: code = NotFound desc = could not find container \"0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1\": container with ID starting with 0837059faa5a2ddf9f99e69be75de99211f3f685b4933ece685b432cb78b91c1 not found: ID does not exist"
Oct 29 22:15:40.666447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666405    4451 scope.go:122] "RemoveContainer" containerID="b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"
Oct 29 22:15:40.666574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666557    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476"} err="failed to get container status \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": rpc error: code = NotFound desc = could not find container \"b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476\": container with ID starting with b4494498226669fc88d01d57b8bbd692975cbf93f6a345eb0aa06bdefa122476 not found: ID does not exist"
Oct 29 22:15:40.666614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666575    4451 scope.go:122] "RemoveContainer" containerID="97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"
Oct 29 22:15:40.666735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.666719    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05"} err="failed to get container status \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": rpc error: code = NotFound desc = could not find container \"97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05\": container with ID starting with 97b320d428f2067cd1b350e65ff5b382e010ea9fd568210bc186bc62206a4b05 not found: ID does not exist"
Oct 29 22:15:40.682570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682538    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:15:40.682805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682793    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="prometheus"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682806    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="prometheus"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682818    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="init-config-reloader"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682823    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="init-config-reloader"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682831    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682836    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682842    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="init-config-reloader"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682846    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="init-config-reloader"
Oct 29 22:15:40.682850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682852    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682857    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682864    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-web"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682868    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-web"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682876    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="prom-label-proxy"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682881    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="prom-label-proxy"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682887    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="config-reloader"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682892    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="config-reloader"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682897    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-web"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682902    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-web"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682924    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="alertmanager"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682930    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="alertmanager"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682936    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="thanos-sidecar"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682940    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="thanos-sidecar"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682951    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b3274c90-4ccc-4830-bc73-a1fd787874db" containerName="console"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682955    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3274c90-4ccc-4830-bc73-a1fd787874db" containerName="console"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682963    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="config-reloader"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682969    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="config-reloader"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682978    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-metric"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682982    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-metric"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682989    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-thanos"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.682993    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-thanos"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683040    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-web"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683049    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="prom-label-proxy"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683054    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="config-reloader"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683061    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-web"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683066    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="alertmanager"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683072    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="thanos-sidecar"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683080    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3274c90-4ccc-4830-bc73-a1fd787874db" containerName="console"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683086    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="config-reloader"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683091    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy-metric"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683096    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy-thanos"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683102    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="prometheus"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683110    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373ca33-9005-45c0-9d0b-a04b6dbc036d" containerName="kube-rbac-proxy"
Oct 29 22:15:40.683209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.683117    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" containerName="kube-rbac-proxy"
Oct 29 22:15:40.693814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.693798    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.708325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.708304    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 22:15:40.708325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.708311    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 22:15:40.709065 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709022    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 22:15:40.709260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709099    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 22:15:40.709260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709148    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 22:15:40.709260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709159    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 22:15:40.709260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709164    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 22:15:40.709463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709279    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 22:15:40.709463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709287    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 22:15:40.709463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709300    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 22:15:40.709463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709284    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 22:15:40.709652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709498    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7xf7\""
Oct 29 22:15:40.709652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709518    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ecd5mvg2qg51g\""
Oct 29 22:15:40.709652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709553    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 22:15:40.709652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.709611    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 22:15:40.710750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.710731    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:15:40.720148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.720122    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 22:15:40.720241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.720196    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 22:15:40.749437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749410    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749440    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-config-out\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749458    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749474    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749615    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749644    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749663    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749711    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749746    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749821    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749862    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-config\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749950    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-web-config\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.749999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749969    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.749992    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xw2f\" (UniqueName: \"kubernetes.io/projected/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-kube-api-access-6xw2f\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.750063    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.750087    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.750101    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.750117    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.750131    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.750185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.750171    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.850790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850767    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.850907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850800    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-config-out\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.850907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850816    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.850907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850833    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.850907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850862    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.850907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850899    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850937    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850960    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.850987    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851091    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851127    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-config\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851158    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-web-config\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851186    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851218    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xw2f\" (UniqueName: \"kubernetes.io/projected/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-kube-api-access-6xw2f\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851270    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851313    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851336    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851354    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851362    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851386    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851415    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.851932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.851496    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.852425 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.852143    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.852573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.852514    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.852573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.852568    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.854537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.854134    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-config-out\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.854537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.854322    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.854537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.854479    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.854740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.854708    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.855332 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.855306    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.855710    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856033    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-web-config\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856138    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856155    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-config\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856163    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856229    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856280    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.856376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.856299    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.857251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.857235    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:40.868214 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:40.868184    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xw2f\" (UniqueName: \"kubernetes.io/projected/a212c3a9-87bf-4ce9-b528-fcfa1076ecde-kube-api-access-6xw2f\") pod \"prometheus-k8s-0\" (UID: \"a212c3a9-87bf-4ce9-b528-fcfa1076ecde\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:41.130361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:41.130291    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:41.488098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:41.488070    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:15:41.490095 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:15:41.490061    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda212c3a9_87bf_4ce9_b528_fcfa1076ecde.slice/crio-feb7bf8226a4fca21de687a9be5a81cd0859f9ca5820bc9663198f837459fa76 WatchSource:0}: Error finding container feb7bf8226a4fca21de687a9be5a81cd0859f9ca5820bc9663198f837459fa76: Status 404 returned error can't find the container with id feb7bf8226a4fca21de687a9be5a81cd0859f9ca5820bc9663198f837459fa76
Oct 29 22:15:41.612441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:41.612408    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"a12b272d445e7fb64d3f5ade481be061daf28e2d11fb6c7cbeb7e3422941f17b"}
Oct 29 22:15:41.612550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:41.612471    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"feb7bf8226a4fca21de687a9be5a81cd0859f9ca5820bc9663198f837459fa76"}
Oct 29 22:15:41.712857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:41.712819    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bfd86cd7c-9wsgc"]
Oct 29 22:15:42.162007 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:42.161976    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7666d69-65dc-42f3-8702-ffb3d664f29b" path="/var/lib/kubelet/pods/b7666d69-65dc-42f3-8702-ffb3d664f29b/volumes"
Oct 29 22:15:42.618271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:42.618236    4451 generic.go:356] "Generic (PLEG): container finished" podID="a212c3a9-87bf-4ce9-b528-fcfa1076ecde" containerID="a12b272d445e7fb64d3f5ade481be061daf28e2d11fb6c7cbeb7e3422941f17b" exitCode=0
Oct 29 22:15:42.618417 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:42.618322    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerDied","Data":"a12b272d445e7fb64d3f5ade481be061daf28e2d11fb6c7cbeb7e3422941f17b"}
Oct 29 22:15:43.625444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.625411    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"d76f6a81fa585b28e94d077302ec4df4456e466687f3456ee80ec4a75cf78a04"}
Oct 29 22:15:43.625825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.625473    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"d77357580241d2d11fff86455e175b531b9cf58c164706a3da7187dc53ddb5f9"}
Oct 29 22:15:43.625825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.625501    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"4a5940bc49ec00fc63d95e6880080b17e10d152118d7ecb5607e2770d4292d6a"}
Oct 29 22:15:43.625825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.625525    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"f7d88fdbb64df0ad1b0d89a0517165298c9e1fb282db5e86d95bf10979c00e45"}
Oct 29 22:15:43.625825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.625554    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"7a7ef804fc6454dae199622bf7b91e967cae0691d8fdc9dd8b3ed401e518ce27"}
Oct 29 22:15:43.625825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.625578    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a212c3a9-87bf-4ce9-b528-fcfa1076ecde","Type":"ContainerStarted","Data":"8786b7722a54b523b414a65a272f9b308d6fdbe2fb76cfb7e2a4721ca4a8b73b"}
Oct 29 22:15:43.697201 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:43.697146    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.668048578 podStartE2EDuration="3.697132885s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.084307ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:43.686467747 +0000 UTC m=+83.629426485" watchObservedRunningTime="2026-10-29 22:15:43.697132885 +0000 UTC m=+83.640091587"
Oct 29 22:15:46.131411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:15:46.131377    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:15:48.738982 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:15:48.738949    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:16:06.732145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:06.732109    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7bfd86cd7c-9wsgc" podUID="f613a96d-ae8d-448b-b772-b417eb6aae71" containerName="console" containerID="cri-o://4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd" gracePeriod=15
Oct 29 22:16:06.962220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:06.962193    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:16:07.057978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.057894    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.057978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.057971    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058071    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058102    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058127    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058174    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058236    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s\" (UniqueName: \"kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058254    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:07.058330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058268    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle\") pod \"f613a96d-ae8d-448b-b772-b417eb6aae71\" (UID: \"f613a96d-ae8d-448b-b772-b417eb6aae71\") "
Oct 29 22:16:07.058330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058274    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:07.058573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058408    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:07.058573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058514    4451 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.058573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058531    4451 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-console-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.058573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058546    4451 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f613a96d-ae8d-448b-b772-b417eb6aae71-tmp\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.058573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058565    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:07.058782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.058755    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:07.060299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.060263    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:07.060405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.060292    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:07.060405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.060366    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s" pod "f613a96d-ae8d-448b-b772-b417eb6aae71" (UID: "f613a96d-ae8d-448b-b772-b417eb6aae71"). InnerVolumeSpecName "kube-api-access-mpg9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:07.159748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.159720    4451 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-oauth-config\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.159881 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.159762    4451 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-oauth-serving-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.159881 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.159772    4451 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f613a96d-ae8d-448b-b772-b417eb6aae71-console-serving-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.159881 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.159780    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpg9s\" (UniqueName: \"kubernetes.io/projected/f613a96d-ae8d-448b-b772-b417eb6aae71-kube-api-access-mpg9s\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.159881 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.159790    4451 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f613a96d-ae8d-448b-b772-b417eb6aae71-trusted-ca-bundle\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.713565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.713529    4451 generic.go:356] "Generic (PLEG): container finished" podID="f613a96d-ae8d-448b-b772-b417eb6aae71" containerID="4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd" exitCode=0
Oct 29 22:16:07.713740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.713597    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfd86cd7c-9wsgc"
Oct 29 22:16:07.713740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.713615    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfd86cd7c-9wsgc" event={"ID":"f613a96d-ae8d-448b-b772-b417eb6aae71","Type":"ContainerDied","Data":"4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd"}
Oct 29 22:16:07.713740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.713673    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfd86cd7c-9wsgc" event={"ID":"f613a96d-ae8d-448b-b772-b417eb6aae71","Type":"ContainerDied","Data":"3249282c3cd0a18a26a0582cbf87c0517b6953b928cbd751b125d89a5093bd89"}
Oct 29 22:16:07.713740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.713693    4451 scope.go:122] "RemoveContainer" containerID="4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd"
Oct 29 22:16:07.722746 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.722730    4451 scope.go:122] "RemoveContainer" containerID="4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd"
Oct 29 22:16:07.723083 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:16:07.723064    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd\": container with ID starting with 4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd not found: ID does not exist" containerID="4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd"
Oct 29 22:16:07.723148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.723093    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd"} err="failed to get container status \"4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd\": rpc error: code = NotFound desc = could not find container \"4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd\": container with ID starting with 4a8ea4537b351b96ff2a2e326df734042064b43bce3d74244fa293371de09acd not found: ID does not exist"
Oct 29 22:16:07.757318 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.757292    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bfd86cd7c-9wsgc"]
Oct 29 22:16:07.767255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:07.767232    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bfd86cd7c-9wsgc"]
Oct 29 22:16:08.160220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:08.160184    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f613a96d-ae8d-448b-b772-b417eb6aae71" path="/var/lib/kubelet/pods/f613a96d-ae8d-448b-b772-b417eb6aae71/volumes"
Oct 29 22:16:18.739356 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:16:18.739308    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:16:41.131125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:41.131043    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:16:41.146319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:41.146294    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:16:41.853405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:16:41.853377    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:16:48.739146 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:16:48.739109    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:17:18.738293 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:17:18.738257    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:17:48.739218 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:17:48.739175    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:18:18.739553 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:18:18.739516    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:18:44.006670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.006629    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"]
Oct 29 22:18:44.009185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.007071    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f613a96d-ae8d-448b-b772-b417eb6aae71" containerName="console"
Oct 29 22:18:44.009185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.007088    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f613a96d-ae8d-448b-b772-b417eb6aae71" containerName="console"
Oct 29 22:18:44.009185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.007174    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f613a96d-ae8d-448b-b772-b417eb6aae71" containerName="console"
Oct 29 22:18:44.010107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.010091    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.042023 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.041998    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"]
Oct 29 22:18:44.110607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.110582    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:18:44.110750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.110729    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:18:44.111017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.110997    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9jtb2\""
Oct 29 22:18:44.111102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.111050    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:18:44.111382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.111366    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:18:44.111426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.111383    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:18:44.165573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.165540    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.165684 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.165615    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.165684 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.165678    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhc8\" (UniqueName: \"kubernetes.io/projected/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-kube-api-access-jhhc8\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.165771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.165716    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.165812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.165783    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.165853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.165822    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266193    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266235    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhc8\" (UniqueName: \"kubernetes.io/projected/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-kube-api-access-jhhc8\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266277    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266314    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266354    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266379    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.266772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.266722    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.268726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.268693    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.268943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.268906    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.269016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.268956    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.269139 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.269124    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.285876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.285854    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhc8\" (UniqueName: \"kubernetes.io/projected/55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa-kube-api-access-jhhc8\") pod \"opendatahub-operator-controller-manager-6f7c578db4-dsx95\" (UID: \"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.326351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.326328    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:44.675369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:44.675344    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"]
Oct 29 22:18:44.677840 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:18:44.677809    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55de3d00_4ac2_45b3_ada2_d4e0c09bb5fa.slice/crio-15b7eab8dad903ae56047751d2f8c15e951d24266f3912579416342891d8a88c WatchSource:0}: Error finding container 15b7eab8dad903ae56047751d2f8c15e951d24266f3912579416342891d8a88c: Status 404 returned error can't find the container with id 15b7eab8dad903ae56047751d2f8c15e951d24266f3912579416342891d8a88c
Oct 29 22:18:45.271634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:45.271594    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95" event={"ID":"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa","Type":"ContainerStarted","Data":"15b7eab8dad903ae56047751d2f8c15e951d24266f3912579416342891d8a88c"}
Oct 29 22:18:48.286356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:48.286270    4451 generic.go:356] "Generic (PLEG): container finished" podID="55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa" containerID="a6f1c6209256c3eec934ec54e43da86b46d238c4d7bce6dc7b22d517f92de703" exitCode=0
Oct 29 22:18:48.286356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:48.286345    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95" event={"ID":"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa","Type":"ContainerDied","Data":"a6f1c6209256c3eec934ec54e43da86b46d238c4d7bce6dc7b22d517f92de703"}
Oct 29 22:18:48.738536 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:18:48.738502    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:18:49.292867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:49.292831    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95" event={"ID":"55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa","Type":"ContainerStarted","Data":"e2efe4f52a89acaf6763fabc9816b7da869f0f5eda0368bb7ef9c5c65d5894a9"}
Oct 29 22:18:49.293335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:49.293077    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:18:49.375724 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:18:49.375674    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95" podStartSLOduration=2.836102322 podStartE2EDuration="6.375658867s" totalImagesPullingTime="3.071753733s" totalInitContainerRuntime="467.802812ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:49.365047931 +0000 UTC m=+269.308006647" watchObservedRunningTime="2026-10-29 22:18:49.375658867 +0000 UTC m=+269.318617572"
Oct 29 22:19:00.298693 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:00.298665    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95"
Oct 29 22:19:04.025602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.025567    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"]
Oct 29 22:19:04.029226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.029208    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.044158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.044136    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:19:04.053975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.053960    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 29 22:19:04.054087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.054059    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b922r\""
Oct 29 22:19:04.054181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.054091    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 29 22:19:04.054181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.054124    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 29 22:19:04.054181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.054139    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 29 22:19:04.057103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.057075    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"]
Oct 29 22:19:04.141191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.141166    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aca505de-81ae-493e-8b00-d68061872026-manager-config\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.141304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.141199    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aca505de-81ae-493e-8b00-d68061872026-cert\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.141304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.141220    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca505de-81ae-493e-8b00-d68061872026-metrics-cert\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.141304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.141278    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk72l\" (UniqueName: \"kubernetes.io/projected/aca505de-81ae-493e-8b00-d68061872026-kube-api-access-bk72l\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.242171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.242134    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aca505de-81ae-493e-8b00-d68061872026-manager-config\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.242312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.242177    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aca505de-81ae-493e-8b00-d68061872026-cert\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.242312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.242200    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca505de-81ae-493e-8b00-d68061872026-metrics-cert\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.242312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.242254    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk72l\" (UniqueName: \"kubernetes.io/projected/aca505de-81ae-493e-8b00-d68061872026-kube-api-access-bk72l\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.242851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.242833    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aca505de-81ae-493e-8b00-d68061872026-manager-config\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.244760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.244731    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca505de-81ae-493e-8b00-d68061872026-metrics-cert\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.244867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.244796    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aca505de-81ae-493e-8b00-d68061872026-cert\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.258899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.258876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk72l\" (UniqueName: \"kubernetes.io/projected/aca505de-81ae-493e-8b00-d68061872026-kube-api-access-bk72l\") pod \"lws-controller-manager-5b88c549dd-hdtld\" (UID: \"aca505de-81ae-493e-8b00-d68061872026\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.340824 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.340801    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:04.632883 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:04.632719    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"]
Oct 29 22:19:04.635676 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:19:04.635649    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca505de_81ae_493e_8b00_d68061872026.slice/crio-6d490ed35e21b08248286c51a331c3be13b4d6b2c768c1416ae0eb96f6adb776 WatchSource:0}: Error finding container 6d490ed35e21b08248286c51a331c3be13b4d6b2c768c1416ae0eb96f6adb776: Status 404 returned error can't find the container with id 6d490ed35e21b08248286c51a331c3be13b4d6b2c768c1416ae0eb96f6adb776
Oct 29 22:19:05.357000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:05.356959    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld" event={"ID":"aca505de-81ae-493e-8b00-d68061872026","Type":"ContainerStarted","Data":"6d490ed35e21b08248286c51a331c3be13b4d6b2c768c1416ae0eb96f6adb776"}
Oct 29 22:19:06.532027 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.531990    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-nzvk9"]
Oct 29 22:19:06.542507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.542480    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:06.555422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.555366    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 29 22:19:06.555674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.555650    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-nzvk9"]
Oct 29 22:19:06.556073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.555707    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-9d8tg\""
Oct 29 22:19:06.663480 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.663440    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-webhook-certs\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:06.663646 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.663589    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5fk\" (UniqueName: \"kubernetes.io/projected/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-kube-api-access-fd5fk\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:06.764410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.764369    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5fk\" (UniqueName: \"kubernetes.io/projected/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-kube-api-access-fd5fk\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:06.764590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.764428    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-webhook-certs\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:06.764590 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:19:06.764568    4451 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 29 22:19:06.764672 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:19:06.764627    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-webhook-certs podName:28ea7ab6-5c97-43db-ba3f-d3ea6552337d nodeName:}" failed. No retries permitted until 2026-10-29 22:19:07.264607136 +0000 UTC m=+287.207565832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-webhook-certs") pod "odh-observability-584f99c788-nzvk9" (UID: "28ea7ab6-5c97-43db-ba3f-d3ea6552337d") : secret "odh-observability-webhook-cert" not found
Oct 29 22:19:06.783698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:06.783632    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5fk\" (UniqueName: \"kubernetes.io/projected/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-kube-api-access-fd5fk\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:07.269461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.269429    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-webhook-certs\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:07.271795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.271769    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/28ea7ab6-5c97-43db-ba3f-d3ea6552337d-webhook-certs\") pod \"odh-observability-584f99c788-nzvk9\" (UID: \"28ea7ab6-5c97-43db-ba3f-d3ea6552337d\") " pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:07.374657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.374610    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld" event={"ID":"aca505de-81ae-493e-8b00-d68061872026","Type":"ContainerStarted","Data":"4031ee075de7b31d33d282586614d86e1a287377b8ff3dab757faa9dacfb6fe0"}
Oct 29 22:19:07.374800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.374711    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:07.460503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.460466    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:07.753240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.753129    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld" podStartSLOduration=2.331923443 podStartE2EDuration="4.75311212s" totalImagesPullingTime="2.421188677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:07.409721625 +0000 UTC m=+287.352680330" watchObservedRunningTime="2026-10-29 22:19:07.75311212 +0000 UTC m=+287.696070839"
Oct 29 22:19:07.754323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:07.754303    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-nzvk9"]
Oct 29 22:19:07.756772 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:19:07.756746    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ea7ab6_5c97_43db_ba3f_d3ea6552337d.slice/crio-562b3902926c9f238cf59f7213e9dabdc897f452a9ea8dae8e2cad1f5e44f3b0 WatchSource:0}: Error finding container 562b3902926c9f238cf59f7213e9dabdc897f452a9ea8dae8e2cad1f5e44f3b0: Status 404 returned error can't find the container with id 562b3902926c9f238cf59f7213e9dabdc897f452a9ea8dae8e2cad1f5e44f3b0
Oct 29 22:19:08.380068 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.380031    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-nzvk9" event={"ID":"28ea7ab6-5c97-43db-ba3f-d3ea6552337d","Type":"ContainerStarted","Data":"562b3902926c9f238cf59f7213e9dabdc897f452a9ea8dae8e2cad1f5e44f3b0"}
Oct 29 22:19:08.855685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.855650    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"]
Oct 29 22:19:08.859576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.859551    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.880201 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.880167    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"]
Oct 29 22:19:08.884791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.884764    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:19:08.884791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.884764    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:19:08.884971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.884797    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:19:08.884971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.884841    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5gxpm\""
Oct 29 22:19:08.989201 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989167    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989211    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpkk\" (UniqueName: \"kubernetes.io/projected/2de4b750-b256-4828-b1a9-d5a5d919c636-kube-api-access-2dpkk\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989235    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989252    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989303    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989325    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989345    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de4b750-b256-4828-b1a9-d5a5d919c636-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989370    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:08.989390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:08.989388    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090195    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090239    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090264    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de4b750-b256-4828-b1a9-d5a5d919c636-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090302    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090322    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090358    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090393    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpkk\" (UniqueName: \"kubernetes.io/projected/2de4b750-b256-4828-b1a9-d5a5d919c636-kube-api-access-2dpkk\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090422    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090448    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090695    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.090841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.090822    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.091118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.091093    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.091199 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.091143    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.091291 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.091266    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de4b750-b256-4828-b1a9-d5a5d919c636-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.093163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.093142    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.093300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.093267    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.107229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.107145    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de4b750-b256-4828-b1a9-d5a5d919c636-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.107329 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.107266    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpkk\" (UniqueName: \"kubernetes.io/projected/2de4b750-b256-4828-b1a9-d5a5d919c636-kube-api-access-2dpkk\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57gktwl\" (UID: \"2de4b750-b256-4828-b1a9-d5a5d919c636\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.177264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.177229    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:09.491644 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:09.491616    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"]
Oct 29 22:19:09.494431 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:19:09.494398    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de4b750_b256_4828_b1a9_d5a5d919c636.slice/crio-8bb6f9c1d11153a2be8f448f72a36080f4caf222f89dd63bbb02ed8c5b82b1d8 WatchSource:0}: Error finding container 8bb6f9c1d11153a2be8f448f72a36080f4caf222f89dd63bbb02ed8c5b82b1d8: Status 404 returned error can't find the container with id 8bb6f9c1d11153a2be8f448f72a36080f4caf222f89dd63bbb02ed8c5b82b1d8
Oct 29 22:19:10.390702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:10.390663    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-nzvk9" event={"ID":"28ea7ab6-5c97-43db-ba3f-d3ea6552337d","Type":"ContainerStarted","Data":"d31afc9996a84d962cc91b66360f8a2aae36772b39041768b996ffda2dedc4db"}
Oct 29 22:19:10.391341 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:10.391322    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:10.392514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:10.392486    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl" event={"ID":"2de4b750-b256-4828-b1a9-d5a5d919c636","Type":"ContainerStarted","Data":"8bb6f9c1d11153a2be8f448f72a36080f4caf222f89dd63bbb02ed8c5b82b1d8"}
Oct 29 22:19:10.431420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:10.431311    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-nzvk9" podStartSLOduration=1.910756576 podStartE2EDuration="4.431295233s" totalImagesPullingTime="2.520538657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:10.429896811 +0000 UTC m=+290.372855517" watchObservedRunningTime="2026-10-29 22:19:10.431295233 +0000 UTC m=+290.374253939"
Oct 29 22:19:12.228621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:12.228586    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:19:12.229086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:12.228673    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:19:12.229086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:12.228722    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:19:12.404354 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:12.404309    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl" event={"ID":"2de4b750-b256-4828-b1a9-d5a5d919c636","Type":"ContainerStarted","Data":"bf1ab93f6b3895facbe5881e286e4c06d75e0c40d26919eb7c51d23cdb40f535"}
Oct 29 22:19:12.460397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:12.460346    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl" podStartSLOduration=1.729614534 podStartE2EDuration="4.460324455s" totalImagesPullingTime="2.730709921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:12.459540127 +0000 UTC m=+292.402498832" watchObservedRunningTime="2026-10-29 22:19:12.460324455 +0000 UTC m=+292.403283160"
Oct 29 22:19:13.177829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:13.177791    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:13.182167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:13.182143    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:13.409358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:13.409322    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:13.410259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:13.410240    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57gktwl"
Oct 29 22:19:18.382302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:18.382273    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-hdtld"
Oct 29 22:19:18.739500 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:19:18.739419    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:19:19.750020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.749989    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"]
Oct 29 22:19:19.753732 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.753713    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.765717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.765697    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gzzsg\""
Oct 29 22:19:19.795903 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.795877    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 29 22:19:19.797202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.797184    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"]
Oct 29 22:19:19.889524 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.889492    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4445af6-ecf4-4b6a-b193-1595743cdcc1-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.889674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.889538    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4445af6-ecf4-4b6a-b193-1595743cdcc1-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.889674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.889604    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4445af6-ecf4-4b6a-b193-1595743cdcc1-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.889757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.889689    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4s5\" (UniqueName: \"kubernetes.io/projected/c4445af6-ecf4-4b6a-b193-1595743cdcc1-kube-api-access-kw4s5\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.990864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.990834    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4s5\" (UniqueName: \"kubernetes.io/projected/c4445af6-ecf4-4b6a-b193-1595743cdcc1-kube-api-access-kw4s5\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.991022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.990988    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4445af6-ecf4-4b6a-b193-1595743cdcc1-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.991094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.991040    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4445af6-ecf4-4b6a-b193-1595743cdcc1-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.991094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.991065    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4445af6-ecf4-4b6a-b193-1595743cdcc1-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.991434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.991412    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4445af6-ecf4-4b6a-b193-1595743cdcc1-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.991557 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.991496    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4445af6-ecf4-4b6a-b193-1595743cdcc1-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:19.992137 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:19.992121    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4445af6-ecf4-4b6a-b193-1595743cdcc1-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:20.007508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:20.007456    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4s5\" (UniqueName: \"kubernetes.io/projected/c4445af6-ecf4-4b6a-b193-1595743cdcc1-kube-api-access-kw4s5\") pod \"kserve-module-controller-manager-5cfb77bf8c-69gvh\" (UID: \"c4445af6-ecf4-4b6a-b193-1595743cdcc1\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:20.066431 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:20.066400    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:20.369582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:20.369551    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"]
Oct 29 22:19:20.372292 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:19:20.372261    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4445af6_ecf4_4b6a_b193_1595743cdcc1.slice/crio-9d0ab33ede49ee362f09b92fdffa287e4a4cf63c41f0eec5ee52b9a293330d27 WatchSource:0}: Error finding container 9d0ab33ede49ee362f09b92fdffa287e4a4cf63c41f0eec5ee52b9a293330d27: Status 404 returned error can't find the container with id 9d0ab33ede49ee362f09b92fdffa287e4a4cf63c41f0eec5ee52b9a293330d27
Oct 29 22:19:20.440749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:20.440720    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh" event={"ID":"c4445af6-ecf4-4b6a-b193-1595743cdcc1","Type":"ContainerStarted","Data":"9d0ab33ede49ee362f09b92fdffa287e4a4cf63c41f0eec5ee52b9a293330d27"}
Oct 29 22:19:22.407556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:22.407524    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-nzvk9"
Oct 29 22:19:23.461119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:23.461083    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh" event={"ID":"c4445af6-ecf4-4b6a-b193-1595743cdcc1","Type":"ContainerStarted","Data":"0fcc157988f8417d70ba407f5901cfb0f6ec9c8bd2116e79c371a042451afb1c"}
Oct 29 22:19:23.461578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:23.461202    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:23.505490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:23.505346    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh" podStartSLOduration=2.1192814269999998 podStartE2EDuration="4.505329509s" totalImagesPullingTime="2.386048082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:23.504865838 +0000 UTC m=+303.447824544" watchObservedRunningTime="2026-10-29 22:19:23.505329509 +0000 UTC m=+303.448288215"
Oct 29 22:19:44.509772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:19:44.509690    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh"
Oct 29 22:19:48.740578 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:19:48.740530    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:20:07.087755 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.087718    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-w6xdh"]
Oct 29 22:20:07.091501 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.091480    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.108102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.108077    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-w6xdh"]
Oct 29 22:20:07.115431 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.115411    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 29 22:20:07.115556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.115491    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pjswg\""
Oct 29 22:20:07.120209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.120183    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"]
Oct 29 22:20:07.123576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.123557    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.144113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.144086    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"]
Oct 29 22:20:07.161578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.161548    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 29 22:20:07.161683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.161595    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-z5dfv\""
Oct 29 22:20:07.165342 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.165321    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-45d6d"]
Oct 29 22:20:07.168756 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.168740    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.197080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.197042    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-45d6d"]
Oct 29 22:20:07.204685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.204663    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4kfs9\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.204788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.204694    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctt97\" (UniqueName: \"kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97\") pod \"llmisvc-controller-manager-869d5dcc5d-4kfs9\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.204788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.204731    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78d3e1ef-babd-4101-a019-853ecab8fe11-cert\") pod \"kserve-controller-manager-776cc66d47-w6xdh\" (UID: \"78d3e1ef-babd-4101-a019-853ecab8fe11\") " pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.204788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.204781    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwgw\" (UniqueName: \"kubernetes.io/projected/78d3e1ef-babd-4101-a019-853ecab8fe11-kube-api-access-9rwgw\") pod \"kserve-controller-manager-776cc66d47-w6xdh\" (UID: \"78d3e1ef-babd-4101-a019-853ecab8fe11\") " pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.209035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.209016    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 29 22:20:07.209104 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.209082    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-p9mrf\""
Oct 29 22:20:07.246092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.246063    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-v8zwm"]
Oct 29 22:20:07.251052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.251030    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.277045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.277021    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-v8zwm"]
Oct 29 22:20:07.305244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.305210    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78d3e1ef-babd-4101-a019-853ecab8fe11-cert\") pod \"kserve-controller-manager-776cc66d47-w6xdh\" (UID: \"78d3e1ef-babd-4101-a019-853ecab8fe11\") " pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.305244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.305244    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwgw\" (UniqueName: \"kubernetes.io/projected/78d3e1ef-babd-4101-a019-853ecab8fe11-kube-api-access-9rwgw\") pod \"kserve-controller-manager-776cc66d47-w6xdh\" (UID: \"78d3e1ef-babd-4101-a019-853ecab8fe11\") " pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.305415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.305265    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2j6\" (UniqueName: \"kubernetes.io/projected/66f78ee2-8da0-4ff3-87f0-25782d169c28-kube-api-access-xc2j6\") pod \"model-serving-api-75bd89fdfc-45d6d\" (UID: \"66f78ee2-8da0-4ff3-87f0-25782d169c28\") " pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.305415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.305353    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4kfs9\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.305415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.305382    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctt97\" (UniqueName: \"kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97\") pod \"llmisvc-controller-manager-869d5dcc5d-4kfs9\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.305568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.305426    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f78ee2-8da0-4ff3-87f0-25782d169c28-tls-certs\") pod \"model-serving-api-75bd89fdfc-45d6d\" (UID: \"66f78ee2-8da0-4ff3-87f0-25782d169c28\") " pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.307780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.307758    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4kfs9\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.307878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.307840    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78d3e1ef-babd-4101-a019-853ecab8fe11-cert\") pod \"kserve-controller-manager-776cc66d47-w6xdh\" (UID: \"78d3e1ef-babd-4101-a019-853ecab8fe11\") " pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.309051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.309033    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 29 22:20:07.309147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.309131    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 29 22:20:07.309185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.309162    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h5j7m\""
Oct 29 22:20:07.321446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.321420    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwgw\" (UniqueName: \"kubernetes.io/projected/78d3e1ef-babd-4101-a019-853ecab8fe11-kube-api-access-9rwgw\") pod \"kserve-controller-manager-776cc66d47-w6xdh\" (UID: \"78d3e1ef-babd-4101-a019-853ecab8fe11\") " pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.321670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.321650    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctt97\" (UniqueName: \"kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97\") pod \"llmisvc-controller-manager-869d5dcc5d-4kfs9\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.406964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.406864    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52684f1d-5db2-4f57-8002-aca7207fb2d4-metrics-certs\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.406964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.406900    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwphj\" (UniqueName: \"kubernetes.io/projected/52684f1d-5db2-4f57-8002-aca7207fb2d4-kube-api-access-kwphj\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.407147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.407053    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f78ee2-8da0-4ff3-87f0-25782d169c28-tls-certs\") pod \"model-serving-api-75bd89fdfc-45d6d\" (UID: \"66f78ee2-8da0-4ff3-87f0-25782d169c28\") " pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.407147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.407113    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2j6\" (UniqueName: \"kubernetes.io/projected/66f78ee2-8da0-4ff3-87f0-25782d169c28-kube-api-access-xc2j6\") pod \"model-serving-api-75bd89fdfc-45d6d\" (UID: \"66f78ee2-8da0-4ff3-87f0-25782d169c28\") " pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.407224 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.407147    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52684f1d-5db2-4f57-8002-aca7207fb2d4-cert\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.407769 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.407746    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:07.409517 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.409497    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f78ee2-8da0-4ff3-87f0-25782d169c28-tls-certs\") pod \"model-serving-api-75bd89fdfc-45d6d\" (UID: \"66f78ee2-8da0-4ff3-87f0-25782d169c28\") " pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.423781 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.423761    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2j6\" (UniqueName: \"kubernetes.io/projected/66f78ee2-8da0-4ff3-87f0-25782d169c28-kube-api-access-xc2j6\") pod \"model-serving-api-75bd89fdfc-45d6d\" (UID: \"66f78ee2-8da0-4ff3-87f0-25782d169c28\") " pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.437470 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.437445    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:07.485123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.481962    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:07.508614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.508573    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52684f1d-5db2-4f57-8002-aca7207fb2d4-metrics-certs\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.508774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.508628    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwphj\" (UniqueName: \"kubernetes.io/projected/52684f1d-5db2-4f57-8002-aca7207fb2d4-kube-api-access-kwphj\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.508774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.508744    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52684f1d-5db2-4f57-8002-aca7207fb2d4-cert\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.511579 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.511552    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52684f1d-5db2-4f57-8002-aca7207fb2d4-cert\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.511741 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.511716    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52684f1d-5db2-4f57-8002-aca7207fb2d4-metrics-certs\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.526454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.526426    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwphj\" (UniqueName: \"kubernetes.io/projected/52684f1d-5db2-4f57-8002-aca7207fb2d4-kube-api-access-kwphj\") pod \"odh-model-controller-f87fcf56c-v8zwm\" (UID: \"52684f1d-5db2-4f57-8002-aca7207fb2d4\") " pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.566855 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.566820    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:07.699946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.699890    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-w6xdh"]
Oct 29 22:20:07.702465 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:20:07.702434    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d3e1ef_babd_4101_a019_853ecab8fe11.slice/crio-258f8cf0188b4e0b79be5b96f3216e3a30f2d97f299168beca88168ed4042b1d WatchSource:0}: Error finding container 258f8cf0188b4e0b79be5b96f3216e3a30f2d97f299168beca88168ed4042b1d: Status 404 returned error can't find the container with id 258f8cf0188b4e0b79be5b96f3216e3a30f2d97f299168beca88168ed4042b1d
Oct 29 22:20:07.703827 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.703808    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:20:07.734235 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.734207    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"]
Oct 29 22:20:07.737628 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:20:07.737597    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0151fe65_690b_4ed2_886f_743571baf027.slice/crio-31bd36dbd32e53816e1d98ceac5806b838917be5f46c6cec5268edf94cdaa231 WatchSource:0}: Error finding container 31bd36dbd32e53816e1d98ceac5806b838917be5f46c6cec5268edf94cdaa231: Status 404 returned error can't find the container with id 31bd36dbd32e53816e1d98ceac5806b838917be5f46c6cec5268edf94cdaa231
Oct 29 22:20:07.781880 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.781843    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-45d6d"]
Oct 29 22:20:07.783997 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:20:07.783974    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f78ee2_8da0_4ff3_87f0_25782d169c28.slice/crio-c5ed02a722216860ce9ac78a3d4658a3e0d88a0d38c849d2bcd67857c990f6d0 WatchSource:0}: Error finding container c5ed02a722216860ce9ac78a3d4658a3e0d88a0d38c849d2bcd67857c990f6d0: Status 404 returned error can't find the container with id c5ed02a722216860ce9ac78a3d4658a3e0d88a0d38c849d2bcd67857c990f6d0
Oct 29 22:20:07.859309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:07.859159    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-v8zwm"]
Oct 29 22:20:07.861489 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:20:07.861464    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52684f1d_5db2_4f57_8002_aca7207fb2d4.slice/crio-0779a3289894a6181a1cba0f2ff58308e35ef4852405b7afc44749a788e71f10 WatchSource:0}: Error finding container 0779a3289894a6181a1cba0f2ff58308e35ef4852405b7afc44749a788e71f10: Status 404 returned error can't find the container with id 0779a3289894a6181a1cba0f2ff58308e35ef4852405b7afc44749a788e71f10
Oct 29 22:20:08.647947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:08.647860    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d" event={"ID":"66f78ee2-8da0-4ff3-87f0-25782d169c28","Type":"ContainerStarted","Data":"c5ed02a722216860ce9ac78a3d4658a3e0d88a0d38c849d2bcd67857c990f6d0"}
Oct 29 22:20:08.650944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:08.650885    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh" event={"ID":"78d3e1ef-babd-4101-a019-853ecab8fe11","Type":"ContainerStarted","Data":"258f8cf0188b4e0b79be5b96f3216e3a30f2d97f299168beca88168ed4042b1d"}
Oct 29 22:20:08.654156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:08.654127    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9" event={"ID":"0151fe65-690b-4ed2-886f-743571baf027","Type":"ContainerStarted","Data":"31bd36dbd32e53816e1d98ceac5806b838917be5f46c6cec5268edf94cdaa231"}
Oct 29 22:20:08.656288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:08.656209    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm" event={"ID":"52684f1d-5db2-4f57-8002-aca7207fb2d4","Type":"ContainerStarted","Data":"0779a3289894a6181a1cba0f2ff58308e35ef4852405b7afc44749a788e71f10"}
Oct 29 22:20:10.671255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:10.671171    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d" event={"ID":"66f78ee2-8da0-4ff3-87f0-25782d169c28","Type":"ContainerStarted","Data":"2f41ae15ca37961a827da80d45a7390dea753d8cd5dad10f5753cb90bd9fd5cc"}
Oct 29 22:20:10.671614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:10.671271    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:10.703979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:10.703903    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d" podStartSLOduration=1.128411845 podStartE2EDuration="3.703889805s" totalImagesPullingTime="2.57547796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:10.70230628 +0000 UTC m=+350.645264999" watchObservedRunningTime="2026-10-29 22:20:10.703889805 +0000 UTC m=+350.646848510"
Oct 29 22:20:11.683679 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.683583    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh" event={"ID":"78d3e1ef-babd-4101-a019-853ecab8fe11","Type":"ContainerStarted","Data":"4f674e897fd657aac0525af9ca6a8721f94ed8e7e4b56d1ad473eea0eaf8f301"}
Oct 29 22:20:11.684139 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.683684    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:11.685289 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.685258    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9" event={"ID":"0151fe65-690b-4ed2-886f-743571baf027","Type":"ContainerStarted","Data":"2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996"}
Oct 29 22:20:11.685409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.685319    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:11.686620 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.686600    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm" event={"ID":"52684f1d-5db2-4f57-8002-aca7207fb2d4","Type":"ContainerStarted","Data":"f69bcd254f7d7b7231e810b503619d2e83dde5ba3868219404434801b22ae70c"}
Oct 29 22:20:11.686712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.686654    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:11.715282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.715237    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh" podStartSLOduration=1.672860428 podStartE2EDuration="4.715226512s" totalImagesPullingTime="3.042366084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:11.714381561 +0000 UTC m=+351.657340266" watchObservedRunningTime="2026-10-29 22:20:11.715226512 +0000 UTC m=+351.658185217"
Oct 29 22:20:11.745351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.745294    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm" podStartSLOduration=1.2211914369999999 podStartE2EDuration="4.745276696s" totalImagesPullingTime="3.524085259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:11.745003144 +0000 UTC m=+351.687961873" watchObservedRunningTime="2026-10-29 22:20:11.745276696 +0000 UTC m=+351.688235401"
Oct 29 22:20:11.776188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:11.776140    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9" podStartSLOduration=1.760748695 podStartE2EDuration="4.776127822s" totalImagesPullingTime="3.015379127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:11.774792058 +0000 UTC m=+351.717750764" watchObservedRunningTime="2026-10-29 22:20:11.776127822 +0000 UTC m=+351.719086527"
Oct 29 22:20:18.742125 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:20:18.742090    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:20:21.691447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:21.691418    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-45d6d"
Oct 29 22:20:22.694571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:22.694543    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-v8zwm"
Oct 29 22:20:42.694022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:42.693990    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:20:42.696774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:20:42.696757    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-w6xdh"
Oct 29 22:20:48.740144 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:20:48.740105    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:21:18.739781 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:21:18.739736    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:21:20.177551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.177522    4451 scope.go:122] "RemoveContainer" containerID="1db8e1158c9eaa4a43a577c4e7f94516d015a303c6261092dd9553a1f0ab80f9"
Oct 29 22:21:20.185036 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.185010    4451 scope.go:122] "RemoveContainer" containerID="40355d60af29031d27cba5e396e7473d51e282a400b54bf3a744a2862a9e7a3e"
Oct 29 22:21:20.191956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.191936    4451 scope.go:122] "RemoveContainer" containerID="6dc1434d31867cda062ce4dee3dbf313f12b4ec6f8adb594fdf471e8a7183f14"
Oct 29 22:21:20.198591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.198571    4451 scope.go:122] "RemoveContainer" containerID="d909bf7f0173a2a6cbcf2f5a64f154dcfd11146ca13176478c54f4c112fe908f"
Oct 29 22:21:20.204905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.204889    4451 scope.go:122] "RemoveContainer" containerID="5a53b9d49efa9078b0f157b0d739836fdb40e95849316b975f93456f4596bb66"
Oct 29 22:21:20.211740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.211722    4451 scope.go:122] "RemoveContainer" containerID="6feb03babe17187d4475e7e4d55eaaac0c825c1fe3955b7fea3a82ee721d9895"
Oct 29 22:21:20.218536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:20.218515    4451 scope.go:122] "RemoveContainer" containerID="a0a60a93d6f91cb98d701692305fd68817b66dc3c231a1087957dc0cb8174b13"
Oct 29 22:21:48.740613 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:21:48.740573    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:21:52.239141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.239106    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"]
Oct 29 22:21:52.244190 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.244168    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.256443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.256416    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"]
Oct 29 22:21:52.264317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.264286    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:21:52.264486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.264466    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:21:52.275534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.275509    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6hs2c\""
Oct 29 22:21:52.403646 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.403616    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6cn\" (UniqueName: \"kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn\") pod \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.403802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.403678    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.505154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.505077    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6cn\" (UniqueName: \"kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn\") pod \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.505154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.505134    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.505472 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.505456    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.522382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.522356    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6cn\" (UniqueName: \"kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn\") pod \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.557256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.557226    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:52.874964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:52.874940    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"]
Oct 29 22:21:52.877161 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:21:52.877135    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3273c80e_3d76_4582_a7bf_c657d345a77b.slice/crio-9244f7ff3a8d52c0d72d54f8cbb6e9330d86380fbdeb69a93f1b9d6074aa329d WatchSource:0}: Error finding container 9244f7ff3a8d52c0d72d54f8cbb6e9330d86380fbdeb69a93f1b9d6074aa329d: Status 404 returned error can't find the container with id 9244f7ff3a8d52c0d72d54f8cbb6e9330d86380fbdeb69a93f1b9d6074aa329d
Oct 29 22:21:53.130833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:53.130753    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv" event={"ID":"3273c80e-3d76-4582-a7bf-c657d345a77b","Type":"ContainerStarted","Data":"9244f7ff3a8d52c0d72d54f8cbb6e9330d86380fbdeb69a93f1b9d6074aa329d"}
Oct 29 22:21:54.428184 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.428141    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"]
Oct 29 22:21:54.430468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.430444    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"]
Oct 29 22:21:54.442003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.441972    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"]
Oct 29 22:21:54.451089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.451065    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.479593 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.471177    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"]
Oct 29 22:21:54.524260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.524221    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2qb2\" (UniqueName: \"kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2\") pod \"kuadrant-operator-controller-manager-6b76784d48-vgtgb\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.524436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.524365    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vgtgb\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.625788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.625746    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2qb2\" (UniqueName: \"kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2\") pod \"kuadrant-operator-controller-manager-6b76784d48-vgtgb\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.625995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.625891    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vgtgb\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.626245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.626224    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vgtgb\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.647403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.647371    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2qb2\" (UniqueName: \"kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2\") pod \"kuadrant-operator-controller-manager-6b76784d48-vgtgb\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:54.769279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:54.769206    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:55.102868 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:55.102835    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"]
Oct 29 22:21:55.107215 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:21:55.107186    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a826460_a88a_4ff7_beb8_57e4e3b67294.slice/crio-805afd51d20fa5c6ffe60887b6236e956a821e8106cfbf0a7bbc0e49a814923c WatchSource:0}: Error finding container 805afd51d20fa5c6ffe60887b6236e956a821e8106cfbf0a7bbc0e49a814923c: Status 404 returned error can't find the container with id 805afd51d20fa5c6ffe60887b6236e956a821e8106cfbf0a7bbc0e49a814923c
Oct 29 22:21:55.144906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:55.144861    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb" event={"ID":"7a826460-a88a-4ff7-beb8-57e4e3b67294","Type":"ContainerStarted","Data":"805afd51d20fa5c6ffe60887b6236e956a821e8106cfbf0a7bbc0e49a814923c"}
Oct 29 22:21:57.157769 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.157729    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" containerName="manager" containerID="cri-o://7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37" gracePeriod=2
Oct 29 22:21:57.410870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.410847    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:57.422044 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:21:57.422011    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:57.557118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.557033    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn\" (UniqueName: \"kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn\") pod \"3273c80e-3d76-4582-a7bf-c657d345a77b\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") "
Oct 29 22:21:57.557279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.557131    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume\") pod \"3273c80e-3d76-4582-a7bf-c657d345a77b\" (UID: \"3273c80e-3d76-4582-a7bf-c657d345a77b\") "
Oct 29 22:21:57.557425 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.557395    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume" pod "3273c80e-3d76-4582-a7bf-c657d345a77b" (UID: "3273c80e-3d76-4582-a7bf-c657d345a77b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:57.559205 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.559183    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn" pod "3273c80e-3d76-4582-a7bf-c657d345a77b" (UID: "3273c80e-3d76-4582-a7bf-c657d345a77b"). InnerVolumeSpecName "kube-api-access-tg6cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:57.658621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.658586    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg6cn\" (UniqueName: \"kubernetes.io/projected/3273c80e-3d76-4582-a7bf-c657d345a77b-kube-api-access-tg6cn\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:21:57.658621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:57.658615    4451 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3273c80e-3d76-4582-a7bf-c657d345a77b-extensions-socket-volume\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:21:58.162590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.162555    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" path="/var/lib/kubelet/pods/3273c80e-3d76-4582-a7bf-c657d345a77b/volumes"
Oct 29 22:21:58.163961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.163926    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb" event={"ID":"7a826460-a88a-4ff7-beb8-57e4e3b67294","Type":"ContainerStarted","Data":"fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d"}
Oct 29 22:21:58.164113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.164063    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:21:58.165322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.165298    4451 generic.go:356] "Generic (PLEG): container finished" podID="3273c80e-3d76-4582-a7bf-c657d345a77b" containerID="7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37" exitCode=2
Oct 29 22:21:58.165428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.165334    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:58.165428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.165341    4451 scope.go:122] "RemoveContainer" containerID="7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37"
Oct 29 22:21:58.174770 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.174755    4451 scope.go:122] "RemoveContainer" containerID="7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37"
Oct 29 22:21:58.175052 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:21:58.175035    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37\": container with ID starting with 7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37 not found: ID does not exist" containerID="7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37"
Oct 29 22:21:58.175100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.175060    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37"} err="failed to get container status \"7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37\": rpc error: code = NotFound desc = could not find container \"7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37\": container with ID starting with 7e44a8c528a3c7056bd70cd1faf87b3236ebf2ad2bcf4262a1f4a31046adcb37 not found: ID does not exist"
Oct 29 22:21:58.230603 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:21:58.230563    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:21:58.230813 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:21:58.230772    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb" podStartSLOduration=2.010279809 podStartE2EDuration="4.230757207s" totalImagesPullingTime="2.220477398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:58.219308021 +0000 UTC m=+458.162266726" watchObservedRunningTime="2026-10-29 22:21:58.230757207 +0000 UTC m=+458.173715912"
Oct 29 22:22:00.169617 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:22:00.169578    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-thpmv\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-thpmv"
Oct 29 22:22:09.173188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:09.173159    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:22:18.739485 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:22:18.739441    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:22:22.256594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.256551    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"]
Oct 29 22:22:22.257115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.256765    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb" podUID="7a826460-a88a-4ff7-beb8-57e4e3b67294" containerName="manager" containerID="cri-o://fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d" gracePeriod=10
Oct 29 22:22:22.499585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.499562    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:22:22.583887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.583858    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume\") pod \"7a826460-a88a-4ff7-beb8-57e4e3b67294\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") "
Oct 29 22:22:22.584055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.583899    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2\" (UniqueName: \"kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2\") pod \"7a826460-a88a-4ff7-beb8-57e4e3b67294\" (UID: \"7a826460-a88a-4ff7-beb8-57e4e3b67294\") "
Oct 29 22:22:22.584205 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.584181    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume" pod "7a826460-a88a-4ff7-beb8-57e4e3b67294" (UID: "7a826460-a88a-4ff7-beb8-57e4e3b67294"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:22:22.585886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.585865    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2" pod "7a826460-a88a-4ff7-beb8-57e4e3b67294" (UID: "7a826460-a88a-4ff7-beb8-57e4e3b67294"). InnerVolumeSpecName "kube-api-access-m2qb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:22:22.685221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.685193    4451 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a826460-a88a-4ff7-beb8-57e4e3b67294-extensions-socket-volume\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:22:22.685221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:22.685220    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2qb2\" (UniqueName: \"kubernetes.io/projected/7a826460-a88a-4ff7-beb8-57e4e3b67294-kube-api-access-m2qb2\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:22:23.272209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.272178    4451 generic.go:356] "Generic (PLEG): container finished" podID="7a826460-a88a-4ff7-beb8-57e4e3b67294" containerID="fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d" exitCode=0
Oct 29 22:22:23.272209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.272212    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb" event={"ID":"7a826460-a88a-4ff7-beb8-57e4e3b67294","Type":"ContainerDied","Data":"fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d"}
Oct 29 22:22:23.272731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.272219    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"
Oct 29 22:22:23.272731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.272248    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb" event={"ID":"7a826460-a88a-4ff7-beb8-57e4e3b67294","Type":"ContainerDied","Data":"805afd51d20fa5c6ffe60887b6236e956a821e8106cfbf0a7bbc0e49a814923c"}
Oct 29 22:22:23.272731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.272269    4451 scope.go:122] "RemoveContainer" containerID="fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d"
Oct 29 22:22:23.280943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.280907    4451 scope.go:122] "RemoveContainer" containerID="fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d"
Oct 29 22:22:23.281211 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:22:23.281190    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d\": container with ID starting with fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d not found: ID does not exist" containerID="fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d"
Oct 29 22:22:23.281282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.281218    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d"} err="failed to get container status \"fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d\": rpc error: code = NotFound desc = could not find container \"fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d\": container with ID starting with fdf3437d2ef093f953a07b193f8531c081628255341b3e1265ee4246f3d3b61d not found: ID does not exist"
Oct 29 22:22:23.327343 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.327319    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"]
Oct 29 22:22:23.337447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:23.337430    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vgtgb"]
Oct 29 22:22:24.160847 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:24.160818    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a826460-a88a-4ff7-beb8-57e4e3b67294" path="/var/lib/kubelet/pods/7a826460-a88a-4ff7-beb8-57e4e3b67294/volumes"
Oct 29 22:22:26.270895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.270859    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"]
Oct 29 22:22:26.271447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.271430    4451 cpu_manager.go:414] "containerMap: removing container" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" containerName="manager"
Oct 29 22:22:26.271493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.271449    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" containerName="manager"
Oct 29 22:22:26.271493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.271465    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7a826460-a88a-4ff7-beb8-57e4e3b67294" containerName="manager"
Oct 29 22:22:26.271493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.271470    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a826460-a88a-4ff7-beb8-57e4e3b67294" containerName="manager"
Oct 29 22:22:26.271582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.271546    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="3273c80e-3d76-4582-a7bf-c657d345a77b" containerName="manager"
Oct 29 22:22:26.271582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.271556    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a826460-a88a-4ff7-beb8-57e4e3b67294" containerName="manager"
Oct 29 22:22:26.276074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.276051    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.292701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.292676    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-67ntj\""
Oct 29 22:22:26.301110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.301081    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"]
Oct 29 22:22:26.318231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318193    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318239    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318299    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318316    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28sj\" (UniqueName: \"kubernetes.io/projected/bb95abec-ee76-4a70-9af9-49ec145c1150-kube-api-access-f28sj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318407    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318452    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318539    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318568    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.318761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.318594    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb95abec-ee76-4a70-9af9-49ec145c1150-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419166    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb95abec-ee76-4a70-9af9-49ec145c1150-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419232    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419263    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419283    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419299    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28sj\" (UniqueName: \"kubernetes.io/projected/bb95abec-ee76-4a70-9af9-49ec145c1150-kube-api-access-f28sj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419321    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419405    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419470    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419498    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419836    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.419902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419872    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.420049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.419950    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.420049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.420033    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.420146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.420071    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb95abec-ee76-4a70-9af9-49ec145c1150-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.421774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.421735    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.421906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.421896    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.435523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.435493    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb95abec-ee76-4a70-9af9-49ec145c1150-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.435664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.435646    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28sj\" (UniqueName: \"kubernetes.io/projected/bb95abec-ee76-4a70-9af9-49ec145c1150-kube-api-access-f28sj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8d8sf\" (UID: \"bb95abec-ee76-4a70-9af9-49ec145c1150\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.591156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.591120    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:26.896936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.896888    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"]
Oct 29 22:22:26.898889 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:22:26.898863    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb95abec_ee76_4a70_9af9_49ec145c1150.slice/crio-19ebedcf0a8c9a9c2a5b398a676996ed350b2cfbe5619f7929872d4c5972938a WatchSource:0}: Error finding container 19ebedcf0a8c9a9c2a5b398a676996ed350b2cfbe5619f7929872d4c5972938a: Status 404 returned error can't find the container with id 19ebedcf0a8c9a9c2a5b398a676996ed350b2cfbe5619f7929872d4c5972938a
Oct 29 22:22:26.901139 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.901100    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:22:26.901208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.901188    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:22:26.901254 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:26.901229    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:22:27.293666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:27.293585    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" event={"ID":"bb95abec-ee76-4a70-9af9-49ec145c1150","Type":"ContainerStarted","Data":"5a7bbe28aeb95ac0b2b53b5412a43da652096d3fc1a08357a8b3786251d2ce02"}
Oct 29 22:22:27.293666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:27.293639    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" event={"ID":"bb95abec-ee76-4a70-9af9-49ec145c1150","Type":"ContainerStarted","Data":"19ebedcf0a8c9a9c2a5b398a676996ed350b2cfbe5619f7929872d4c5972938a"}
Oct 29 22:22:27.353036 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:27.352976    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podStartSLOduration=1.352959128 podStartE2EDuration="1.352959128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:27.342426942 +0000 UTC m=+487.285385647" watchObservedRunningTime="2026-10-29 22:22:27.352959128 +0000 UTC m=+487.295917833"
Oct 29 22:22:27.591677 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:27.591651    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:27.596729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:27.596705    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:28.298200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:28.298163    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:28.299163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:28.299138    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:22:40.076054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.075975    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:40.079398 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.079382    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.093587 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.093552    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:22:40.095110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.095084    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:40.103280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.103243    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:22:40.103280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.103253    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qwmm7\""
Oct 29 22:22:40.103449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.103300    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 29 22:22:40.139340 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.139314    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file\") pod \"limitador-limitador-74dfcfb894-lzb7m\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.139457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.139364    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjx2\" (UniqueName: \"kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2\") pod \"limitador-limitador-74dfcfb894-lzb7m\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.204520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.204489    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:40.240251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.240217    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjx2\" (UniqueName: \"kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2\") pod \"limitador-limitador-74dfcfb894-lzb7m\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.240548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.240522    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file\") pod \"limitador-limitador-74dfcfb894-lzb7m\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.241056 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.241038    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file\") pod \"limitador-limitador-74dfcfb894-lzb7m\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.256454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.256431    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjx2\" (UniqueName: \"kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2\") pod \"limitador-limitador-74dfcfb894-lzb7m\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.392615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.392538    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:40.682995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:40.682967    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:40.685456 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:22:40.685426    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6feb18_41ed_4599_b98e_1f45203433e4.slice/crio-0d559874e0900c91709ac751d23a36f4bbc2dfe18150eaacbae1846935bd714f WatchSource:0}: Error finding container 0d559874e0900c91709ac751d23a36f4bbc2dfe18150eaacbae1846935bd714f: Status 404 returned error can't find the container with id 0d559874e0900c91709ac751d23a36f4bbc2dfe18150eaacbae1846935bd714f
Oct 29 22:22:41.353429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:41.353395    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m" event={"ID":"1d6feb18-41ed-4599-b98e-1f45203433e4","Type":"ContainerStarted","Data":"0d559874e0900c91709ac751d23a36f4bbc2dfe18150eaacbae1846935bd714f"}
Oct 29 22:22:42.358731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:42.358692    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m" event={"ID":"1d6feb18-41ed-4599-b98e-1f45203433e4","Type":"ContainerStarted","Data":"b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1"}
Oct 29 22:22:42.359195 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:42.358814    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:42.392645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:42.392603    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m" podStartSLOduration=1.304474312 podStartE2EDuration="2.39259127s" totalImagesPullingTime="1.088116958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:42.391392996 +0000 UTC m=+502.334351702" watchObservedRunningTime="2026-10-29 22:22:42.39259127 +0000 UTC m=+502.335549976"
Oct 29 22:22:48.738642 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:22:48.738610    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:22:53.364339 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:53.364302    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:54.363579 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:54.363546    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:54.363771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:54.363744    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m" podUID="1d6feb18-41ed-4599-b98e-1f45203433e4" containerName="limitador" containerID="cri-o://b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1" gracePeriod=30
Oct 29 22:22:54.910839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:54.910814    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:55.070876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.070851    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file\" (UniqueName: \"kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file\") pod \"1d6feb18-41ed-4599-b98e-1f45203433e4\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") "
Oct 29 22:22:55.071039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.070955    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2\" (UniqueName: \"kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2\") pod \"1d6feb18-41ed-4599-b98e-1f45203433e4\" (UID: \"1d6feb18-41ed-4599-b98e-1f45203433e4\") "
Oct 29 22:22:55.071210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.071189    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file" pod "1d6feb18-41ed-4599-b98e-1f45203433e4" (UID: "1d6feb18-41ed-4599-b98e-1f45203433e4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:22:55.072941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.072905    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2" pod "1d6feb18-41ed-4599-b98e-1f45203433e4" (UID: "1d6feb18-41ed-4599-b98e-1f45203433e4"). InnerVolumeSpecName "kube-api-access-qbjx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:22:55.171999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.171974    4451 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6feb18-41ed-4599-b98e-1f45203433e4-config-file\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:22:55.171999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.171999    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjx2\" (UniqueName: \"kubernetes.io/projected/1d6feb18-41ed-4599-b98e-1f45203433e4-kube-api-access-qbjx2\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:22:55.414689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.414610    4451 generic.go:356] "Generic (PLEG): container finished" podID="1d6feb18-41ed-4599-b98e-1f45203433e4" containerID="b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1" exitCode=0
Oct 29 22:22:55.414689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.414654    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"
Oct 29 22:22:55.414689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.414674    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m" event={"ID":"1d6feb18-41ed-4599-b98e-1f45203433e4","Type":"ContainerDied","Data":"b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1"}
Oct 29 22:22:55.414989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.414725    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lzb7m" event={"ID":"1d6feb18-41ed-4599-b98e-1f45203433e4","Type":"ContainerDied","Data":"0d559874e0900c91709ac751d23a36f4bbc2dfe18150eaacbae1846935bd714f"}
Oct 29 22:22:55.414989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.414753    4451 scope.go:122] "RemoveContainer" containerID="b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1"
Oct 29 22:22:55.432488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.432463    4451 scope.go:122] "RemoveContainer" containerID="b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1"
Oct 29 22:22:55.433018 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:22:55.432994    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1\": container with ID starting with b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1 not found: ID does not exist" containerID="b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1"
Oct 29 22:22:55.433119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.433025    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1"} err="failed to get container status \"b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1\": rpc error: code = NotFound desc = could not find container \"b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1\": container with ID starting with b8f20688c015912da85c0e5a20c86d391f03ba5221e663a54279682299e0eec1 not found: ID does not exist"
Oct 29 22:22:55.459874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.459854    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:55.469814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:55.469791    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lzb7m"]
Oct 29 22:22:56.161719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:22:56.161689    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6feb18-41ed-4599-b98e-1f45203433e4" path="/var/lib/kubelet/pods/1d6feb18-41ed-4599-b98e-1f45203433e4/volumes"
Oct 29 22:23:01.576521 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.576487    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"]
Oct 29 22:23:01.577078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.576902    4451 cpu_manager.go:414] "containerMap: removing container" podUID="1d6feb18-41ed-4599-b98e-1f45203433e4" containerName="limitador"
Oct 29 22:23:01.577078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.576928    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6feb18-41ed-4599-b98e-1f45203433e4" containerName="limitador"
Oct 29 22:23:01.577078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.577008    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6feb18-41ed-4599-b98e-1f45203433e4" containerName="limitador"
Oct 29 22:23:01.581403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.581383    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.595616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.595588    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"]
Oct 29 22:23:01.596787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.596765    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:23:01.596787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.596765    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:23:01.623404 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.623366    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgr4\" (UniqueName: \"kubernetes.io/projected/3783398a-165b-4c51-8af9-2ba445dada6b-kube-api-access-skgr4\") pod \"postgres-7cdbf6dc44-blj7f\" (UID: \"3783398a-165b-4c51-8af9-2ba445dada6b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.623565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.623530    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3783398a-165b-4c51-8af9-2ba445dada6b-data\") pod \"postgres-7cdbf6dc44-blj7f\" (UID: \"3783398a-165b-4c51-8af9-2ba445dada6b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.668786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.668756    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dl95k\""
Oct 29 22:23:01.668786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.668757    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 29 22:23:01.724779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.724746    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgr4\" (UniqueName: \"kubernetes.io/projected/3783398a-165b-4c51-8af9-2ba445dada6b-kube-api-access-skgr4\") pod \"postgres-7cdbf6dc44-blj7f\" (UID: \"3783398a-165b-4c51-8af9-2ba445dada6b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.724974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.724875    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3783398a-165b-4c51-8af9-2ba445dada6b-data\") pod \"postgres-7cdbf6dc44-blj7f\" (UID: \"3783398a-165b-4c51-8af9-2ba445dada6b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.725264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.725243    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3783398a-165b-4c51-8af9-2ba445dada6b-data\") pod \"postgres-7cdbf6dc44-blj7f\" (UID: \"3783398a-165b-4c51-8af9-2ba445dada6b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.741764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.741732    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgr4\" (UniqueName: \"kubernetes.io/projected/3783398a-165b-4c51-8af9-2ba445dada6b-kube-api-access-skgr4\") pod \"postgres-7cdbf6dc44-blj7f\" (UID: \"3783398a-165b-4c51-8af9-2ba445dada6b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:01.894197 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:01.894118    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:02.186374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:02.186349    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"]
Oct 29 22:23:02.188415 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:23:02.188386    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3783398a_165b_4c51_8af9_2ba445dada6b.slice/crio-005d7045cb7cdc2f8e14f2d29d3eccc9e971eec1010faa38b7c879fb8791d2cd WatchSource:0}: Error finding container 005d7045cb7cdc2f8e14f2d29d3eccc9e971eec1010faa38b7c879fb8791d2cd: Status 404 returned error can't find the container with id 005d7045cb7cdc2f8e14f2d29d3eccc9e971eec1010faa38b7c879fb8791d2cd
Oct 29 22:23:02.446470 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:02.446389    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f" event={"ID":"3783398a-165b-4c51-8af9-2ba445dada6b","Type":"ContainerStarted","Data":"005d7045cb7cdc2f8e14f2d29d3eccc9e971eec1010faa38b7c879fb8791d2cd"}
Oct 29 22:23:08.472437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:08.472394    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f" event={"ID":"3783398a-165b-4c51-8af9-2ba445dada6b","Type":"ContainerStarted","Data":"779c519e5af722922f14163f1c39e02dcd82a9d766dbff864aaba14f0cd1fad9"}
Oct 29 22:23:08.472807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:08.472456    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:08.505516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:08.505460    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f" podStartSLOduration=2.163603888 podStartE2EDuration="7.505448391s" totalImagesPullingTime="5.341844503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:08.504706412 +0000 UTC m=+528.447665120" watchObservedRunningTime="2026-10-29 22:23:08.505448391 +0000 UTC m=+528.448407095"
Oct 29 22:23:14.506736 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:14.506709    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-blj7f"
Oct 29 22:23:18.740328 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:23:18.740285    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:23:24.817950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.817282    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-895nw"]
Oct 29 22:23:24.842889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.842859    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-895nw"]
Oct 29 22:23:24.843054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.843007    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:24.865009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.864984    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 22:23:24.865009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.864984    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 22:23:24.865191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.865023    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 22:23:24.865191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.865035    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x42df\""
Oct 29 22:23:24.945905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.945873    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:24.946111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.946035    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:24.946166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:24.946118    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9zd\" (UniqueName: \"kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.046894 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.046851    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.047103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.046936    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9zd\" (UniqueName: \"kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.047103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.046959    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.049331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.049303    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.049446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.049386    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.063098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.063075    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9zd\" (UniqueName: \"kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd\") pod \"maas-controller-577f596567-895nw\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") " pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.159085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.159052    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:25.200220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.200183    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-g9wkx"]
Oct 29 22:23:25.204260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.204236    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.233721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.233682    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-g9wkx"]
Oct 29 22:23:25.350178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.350139    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gg8\" (UniqueName: \"kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.350370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.350263    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.350370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.350336    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.450949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.450846    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gg8\" (UniqueName: \"kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.450949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.450931    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.451169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.451008    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.452570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.452549    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-895nw"]
Oct 29 22:23:25.453656 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.453634    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.453863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.453842    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.454769 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:23:25.454748    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef9c0b2_16a0_4f47_a8bd_da7a61f2b064.slice/crio-068831193c89ea3b1d683c08cad8f2a2c7584ad6831841a885c2d32396004371 WatchSource:0}: Error finding container 068831193c89ea3b1d683c08cad8f2a2c7584ad6831841a885c2d32396004371: Status 404 returned error can't find the container with id 068831193c89ea3b1d683c08cad8f2a2c7584ad6831841a885c2d32396004371
Oct 29 22:23:25.467255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.467236    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gg8\" (UniqueName: \"kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8\") pod \"maas-controller-6c5b76ff8b-g9wkx\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") " pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.530789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.530757    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:25.550897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.550872    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-895nw" event={"ID":"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064","Type":"ContainerStarted","Data":"068831193c89ea3b1d683c08cad8f2a2c7584ad6831841a885c2d32396004371"}
Oct 29 22:23:25.824120 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:25.824090    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-g9wkx"]
Oct 29 22:23:25.826082 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:23:25.826053    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0af4f9_4d1d_439b_b18f_723efd65ec59.slice/crio-e37716798b98541a156907bbc496fba331942587fcdb3c021983704b8d8ee145 WatchSource:0}: Error finding container e37716798b98541a156907bbc496fba331942587fcdb3c021983704b8d8ee145: Status 404 returned error can't find the container with id e37716798b98541a156907bbc496fba331942587fcdb3c021983704b8d8ee145
Oct 29 22:23:26.568748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:26.568707    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx" event={"ID":"6f0af4f9-4d1d-439b-b18f-723efd65ec59","Type":"ContainerStarted","Data":"e37716798b98541a156907bbc496fba331942587fcdb3c021983704b8d8ee145"}
Oct 29 22:23:29.584990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:29.584948    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-895nw" event={"ID":"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064","Type":"ContainerStarted","Data":"b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8"}
Oct 29 22:23:29.585457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:29.585099    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:29.586626 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:29.586602    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx" event={"ID":"6f0af4f9-4d1d-439b-b18f-723efd65ec59","Type":"ContainerStarted","Data":"df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9"}
Oct 29 22:23:29.586750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:29.586728    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:29.623646 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:29.623601    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-895nw" podStartSLOduration=2.550930569 podStartE2EDuration="5.623589629s" totalImagesPullingTime="3.07265906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:29.623123145 +0000 UTC m=+549.566081854" watchObservedRunningTime="2026-10-29 22:23:29.623589629 +0000 UTC m=+549.566548334"
Oct 29 22:23:29.660482 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:29.660427    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx" podStartSLOduration=1.855193596 podStartE2EDuration="4.660414676s" totalImagesPullingTime="2.80522108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:29.659057114 +0000 UTC m=+549.602015819" watchObservedRunningTime="2026-10-29 22:23:29.660414676 +0000 UTC m=+549.603373380"
Oct 29 22:23:32.016094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.016055    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"]
Oct 29 22:23:32.019878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.019856    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.034266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.034221    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-tjzjt\""
Oct 29 22:23:32.036422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.036403    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"]
Oct 29 22:23:32.044293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.044246    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 29 22:23:32.044435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.044277    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 29 22:23:32.044435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.044334    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 29 22:23:32.115433 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.115397    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.115609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.115496    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.115609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.115572    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tct4g\" (UniqueName: \"kubernetes.io/projected/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-kube-api-access-tct4g\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.115609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.115600    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.216870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.216835    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.217070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.216883    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.217070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.216985    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tct4g\" (UniqueName: \"kubernetes.io/projected/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-kube-api-access-tct4g\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.217070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.217016    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.217070 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:23:32.217016    4451 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 29 22:23:32.217323 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:23:32.217091    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-tls-cert podName:d3912e35-99c2-41d6-bae8-f4b0d0d2742a nodeName:}" failed. No retries permitted until 2026-10-29 22:23:32.717072214 +0000 UTC m=+552.660030898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-cvxs6" (UID: "d3912e35-99c2-41d6-bae8-f4b0d0d2742a") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 29 22:23:32.217608 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.217590    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.217786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.217769    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.234622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.234599    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tct4g\" (UniqueName: \"kubernetes.io/projected/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-kube-api-access-tct4g\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.721320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.721287    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.723665 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.723645    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3912e35-99c2-41d6-bae8-f4b0d0d2742a-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-cvxs6\" (UID: \"d3912e35-99c2-41d6-bae8-f4b0d0d2742a\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:32.932396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:32.932357    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:33.224362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:33.224335    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"]
Oct 29 22:23:33.226374 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:23:33.226348    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3912e35_99c2_41d6_bae8_f4b0d0d2742a.slice/crio-4e3dd27599b61af0c7853da3c0efb9264c6d2fa7bb8a6830dcc6c4b4c3e43863 WatchSource:0}: Error finding container 4e3dd27599b61af0c7853da3c0efb9264c6d2fa7bb8a6830dcc6c4b4c3e43863: Status 404 returned error can't find the container with id 4e3dd27599b61af0c7853da3c0efb9264c6d2fa7bb8a6830dcc6c4b4c3e43863
Oct 29 22:23:33.605642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:33.605602    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6" event={"ID":"d3912e35-99c2-41d6-bae8-f4b0d0d2742a","Type":"ContainerStarted","Data":"4e3dd27599b61af0c7853da3c0efb9264c6d2fa7bb8a6830dcc6c4b4c3e43863"}
Oct 29 22:23:40.594230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:40.594197    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:40.595098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:40.595077    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 22:23:40.691579 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:40.691542    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-895nw"]
Oct 29 22:23:40.691853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:40.691826    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-895nw" podUID="fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" containerName="manager" containerID="cri-o://b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8" gracePeriod=30
Oct 29 22:23:40.956444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:40.956418    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:41.101935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.101876    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd\" (UniqueName: \"kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd\") pod \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") "
Oct 29 22:23:41.102114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.101974    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert\") pod \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") "
Oct 29 22:23:41.102114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.102008    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs\") pod \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\" (UID: \"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064\") "
Oct 29 22:23:41.104583 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.104551    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs" pod "fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" (UID: "fef9c0b2-16a0-4f47-a8bd-da7a61f2b064"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:23:41.104730 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.104663    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert" pod "fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" (UID: "fef9c0b2-16a0-4f47-a8bd-da7a61f2b064"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:23:41.105124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.105092    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd" pod "fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" (UID: "fef9c0b2-16a0-4f47-a8bd-da7a61f2b064"). InnerVolumeSpecName "kube-api-access-bk9zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:23:41.202749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.202671    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk9zd\" (UniqueName: \"kubernetes.io/projected/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-kube-api-access-bk9zd\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:23:41.202749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.202702    4451 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-webhook-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:23:41.202749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.202715    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:23:41.652803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.652761    4451 generic.go:356] "Generic (PLEG): container finished" podID="fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" containerID="b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8" exitCode=0
Oct 29 22:23:41.652803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.652804    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-895nw"
Oct 29 22:23:41.653345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.652848    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-895nw" event={"ID":"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064","Type":"ContainerDied","Data":"b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8"}
Oct 29 22:23:41.653345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.652905    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-895nw" event={"ID":"fef9c0b2-16a0-4f47-a8bd-da7a61f2b064","Type":"ContainerDied","Data":"068831193c89ea3b1d683c08cad8f2a2c7584ad6831841a885c2d32396004371"}
Oct 29 22:23:41.653345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.652946    4451 scope.go:122] "RemoveContainer" containerID="b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8"
Oct 29 22:23:41.663408 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.663384    4451 scope.go:122] "RemoveContainer" containerID="b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8"
Oct 29 22:23:41.663723 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:23:41.663697    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8\": container with ID starting with b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8 not found: ID does not exist" containerID="b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8"
Oct 29 22:23:41.663813 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.663729    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8"} err="failed to get container status \"b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8\": rpc error: code = NotFound desc = could not find container \"b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8\": container with ID starting with b856b382a4e13d93500b701e90cb84115dd243ad1cf9157664b03162967c99c8 not found: ID does not exist"
Oct 29 22:23:41.697164 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.697140    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-895nw"]
Oct 29 22:23:41.707310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:41.707290    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-895nw"]
Oct 29 22:23:42.166275 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:42.166243    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" path="/var/lib/kubelet/pods/fef9c0b2-16a0-4f47-a8bd-da7a61f2b064/volumes"
Oct 29 22:23:43.664518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:43.664486    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6" event={"ID":"d3912e35-99c2-41d6-bae8-f4b0d0d2742a","Type":"ContainerStarted","Data":"bab21503e8b207cbf9974917414cc2ff477369675c1b3bab03361f72746d589c"}
Oct 29 22:23:43.664961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:43.664617    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:23:43.698906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:43.698855    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6" podStartSLOduration=2.657692063 podStartE2EDuration="12.698842654s" totalImagesPullingTime="10.041150591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:43.698242811 +0000 UTC m=+563.641201539" watchObservedRunningTime="2026-10-29 22:23:43.698842654 +0000 UTC m=+563.641801353"
Oct 29 22:23:48.739891 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:23:48.739858    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:23:54.674159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:23:54.674128    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6"
Oct 29 22:24:02.456545 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.456510    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"]
Oct 29 22:24:02.456943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.456901    4451 cpu_manager.go:414] "containerMap: removing container" podUID="fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" containerName="manager"
Oct 29 22:24:02.456943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.456926    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" containerName="manager"
Oct 29 22:24:02.457033 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.456990    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef9c0b2-16a0-4f47-a8bd-da7a61f2b064" containerName="manager"
Oct 29 22:24:02.554713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.554677    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"]
Oct 29 22:24:02.554887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.554786    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.567408 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.567384    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 22:24:02.577442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.577421    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:24:02.577672 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.577655    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 22:24:02.586569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.586550    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.586637 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.586601    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbd7\" (UniqueName: \"kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.586740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.586727    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.586787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.586746    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.688247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.688211    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.688247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.688249    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.688466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.688276    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.688466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.688323    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbd7\" (UniqueName: \"kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.689057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.689036    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.690799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.690773    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.690877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.690850    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.718321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.718268    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbd7\" (UniqueName: \"kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7\") pod \"maas-api-5c48877c75-gngtf\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:02.875866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:02.875833    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:03.206393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:03.206365    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"]
Oct 29 22:24:03.208162 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:24:03.208132    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83af32f_c131_47ce_9e0d_48d3cd7391f7.slice/crio-8e5dee8dcc84e44198184719509a6bfd4419d93b15c7b2f99aaa28f991a1d7fa WatchSource:0}: Error finding container 8e5dee8dcc84e44198184719509a6bfd4419d93b15c7b2f99aaa28f991a1d7fa: Status 404 returned error can't find the container with id 8e5dee8dcc84e44198184719509a6bfd4419d93b15c7b2f99aaa28f991a1d7fa
Oct 29 22:24:03.754147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:03.754115    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf" event={"ID":"c83af32f-c131-47ce-9e0d-48d3cd7391f7","Type":"ContainerStarted","Data":"8e5dee8dcc84e44198184719509a6bfd4419d93b15c7b2f99aaa28f991a1d7fa"}
Oct 29 22:24:05.765376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:05.765332    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf" event={"ID":"c83af32f-c131-47ce-9e0d-48d3cd7391f7","Type":"ContainerStarted","Data":"15c5412caae8c3259c48d91ce6b7ead482be5f0703f7b3206eb07f016ea63bf9"}
Oct 29 22:24:05.765857 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:05.765462    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:11.774903 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:11.774873    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 22:24:11.810704 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:11.810660    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf" podStartSLOduration=7.793315592 podStartE2EDuration="9.810647821s" totalImagesPullingTime="2.017332229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:05.811426216 +0000 UTC m=+585.754384929" watchObservedRunningTime="2026-10-29 22:24:11.810647821 +0000 UTC m=+591.753606526"
Oct 29 22:24:12.931618 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:12.931581    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"]
Oct 29 22:24:12.945710 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:12.945689    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:12.951381 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:12.951354    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"]
Oct 29 22:24:12.985131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:12.985100    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aca75a-d414-4403-b0ef-1a42ec9bf04d-cert\") pod \"llmisvc-controller-manager-6466c554d8-cbglz\" (UID: \"99aca75a-d414-4403-b0ef-1a42ec9bf04d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:12.985272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:12.985217    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbngv\" (UniqueName: \"kubernetes.io/projected/99aca75a-d414-4403-b0ef-1a42ec9bf04d-kube-api-access-nbngv\") pod \"llmisvc-controller-manager-6466c554d8-cbglz\" (UID: \"99aca75a-d414-4403-b0ef-1a42ec9bf04d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:13.086123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.086083    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aca75a-d414-4403-b0ef-1a42ec9bf04d-cert\") pod \"llmisvc-controller-manager-6466c554d8-cbglz\" (UID: \"99aca75a-d414-4403-b0ef-1a42ec9bf04d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:13.086294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.086164    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbngv\" (UniqueName: \"kubernetes.io/projected/99aca75a-d414-4403-b0ef-1a42ec9bf04d-kube-api-access-nbngv\") pod \"llmisvc-controller-manager-6466c554d8-cbglz\" (UID: \"99aca75a-d414-4403-b0ef-1a42ec9bf04d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:13.088467 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.088441    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aca75a-d414-4403-b0ef-1a42ec9bf04d-cert\") pod \"llmisvc-controller-manager-6466c554d8-cbglz\" (UID: \"99aca75a-d414-4403-b0ef-1a42ec9bf04d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:13.104185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.104158    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbngv\" (UniqueName: \"kubernetes.io/projected/99aca75a-d414-4403-b0ef-1a42ec9bf04d-kube-api-access-nbngv\") pod \"llmisvc-controller-manager-6466c554d8-cbglz\" (UID: \"99aca75a-d414-4403-b0ef-1a42ec9bf04d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:13.261119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.261036    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:13.388084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.388044    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:39418->10.133.0.33:15021: read: connection reset by peer" start-of-body=
Oct 29 22:24:13.388426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.388117    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podUID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:39418->10.133.0.33:15021: read: connection reset by peer"
Oct 29 22:24:13.557284 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.557169    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"]
Oct 29 22:24:13.559632 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:24:13.559603    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99aca75a_d414_4403_b0ef_1a42ec9bf04d.slice/crio-57c8ff65a1d3765cb70da9dd00d13d5388d1d7c35cc0a64feac28bf05add31c4 WatchSource:0}: Error finding container 57c8ff65a1d3765cb70da9dd00d13d5388d1d7c35cc0a64feac28bf05add31c4: Status 404 returned error can't find the container with id 57c8ff65a1d3765cb70da9dd00d13d5388d1d7c35cc0a64feac28bf05add31c4
Oct 29 22:24:13.803073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.803038    4451 generic.go:356] "Generic (PLEG): container finished" podID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerID="5a7bbe28aeb95ac0b2b53b5412a43da652096d3fc1a08357a8b3786251d2ce02" exitCode=0
Oct 29 22:24:13.803252 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.803108    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" event={"ID":"bb95abec-ee76-4a70-9af9-49ec145c1150","Type":"ContainerDied","Data":"5a7bbe28aeb95ac0b2b53b5412a43da652096d3fc1a08357a8b3786251d2ce02"}
Oct 29 22:24:13.803545 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.803522    4451 scope.go:122] "RemoveContainer" containerID="5a7bbe28aeb95ac0b2b53b5412a43da652096d3fc1a08357a8b3786251d2ce02"
Oct 29 22:24:13.804471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.804449    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz" event={"ID":"99aca75a-d414-4403-b0ef-1a42ec9bf04d","Type":"ContainerStarted","Data":"57c8ff65a1d3765cb70da9dd00d13d5388d1d7c35cc0a64feac28bf05add31c4"}
Oct 29 22:24:13.804716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.804686    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:24:13.804793 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.804759    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:24:13.804838 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:13.804797    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:24:14.819482 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:14.819446    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" event={"ID":"bb95abec-ee76-4a70-9af9-49ec145c1150","Type":"ContainerStarted","Data":"44d0b3424e5943035cf9fa67fe89843cb3777691cb1beba53a16df31090a43be"}
Oct 29 22:24:14.821296 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:14.821264    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz" event={"ID":"99aca75a-d414-4403-b0ef-1a42ec9bf04d","Type":"ContainerStarted","Data":"8c2801720c8ba7bacbdc33f478e18e1560eb5d7520a05471d0880c73c14e9041"}
Oct 29 22:24:14.821430 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:14.821330    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:14.916814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:14.916753    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz" podStartSLOduration=2.559569382 podStartE2EDuration="2.916738167s" totalImagesPullingTime="357.168785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:14.91540578 +0000 UTC m=+594.858364484" watchObservedRunningTime="2026-10-29 22:24:14.916738167 +0000 UTC m=+594.859696873"
Oct 29 22:24:15.596941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:15.596244    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:24:15.596941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:15.596375    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:24:15.596941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:15.596427    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podUID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:24:16.596945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:16.596276    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:24:16.596945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:16.596349    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podUID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:24:17.612935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:17.610335    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:24:17.612935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:17.610398    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podUID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:24:18.596769 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:18.596330    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:24:18.596769 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:18.596396    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podUID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:24:18.740810 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:24:18.740778    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:24:19.592207 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:19.592158    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:24:19.592395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:19.592227    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf" podUID="bb95abec-ee76-4a70-9af9-49ec145c1150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:24:20.595072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:20.595041    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:24:20.595474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:20.595298    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:24:20.595928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:20.595895    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8d8sf"
Oct 29 22:24:45.827156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:45.827121    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cbglz"
Oct 29 22:24:45.886519 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:45.886486    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"]
Oct 29 22:24:45.886736 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:45.886709    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9" podUID="0151fe65-690b-4ed2-886f-743571baf027" containerName="manager" containerID="cri-o://2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996" gracePeriod=30
Oct 29 22:24:46.134244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.134216    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:24:46.304129 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.304099    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97\" (UniqueName: \"kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97\") pod \"0151fe65-690b-4ed2-886f-743571baf027\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") "
Oct 29 22:24:46.304337 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.304155    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert\" (UniqueName: \"kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert\") pod \"0151fe65-690b-4ed2-886f-743571baf027\" (UID: \"0151fe65-690b-4ed2-886f-743571baf027\") "
Oct 29 22:24:46.306293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.306265    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97" pod "0151fe65-690b-4ed2-886f-743571baf027" (UID: "0151fe65-690b-4ed2-886f-743571baf027"). InnerVolumeSpecName "kube-api-access-ctt97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:24:46.306293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.306282    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert" pod "0151fe65-690b-4ed2-886f-743571baf027" (UID: "0151fe65-690b-4ed2-886f-743571baf027"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:24:46.405115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.405053    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctt97\" (UniqueName: \"kubernetes.io/projected/0151fe65-690b-4ed2-886f-743571baf027-kube-api-access-ctt97\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:24:46.405115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.405078    4451 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0151fe65-690b-4ed2-886f-743571baf027-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:24:46.964239 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.964205    4451 generic.go:356] "Generic (PLEG): container finished" podID="0151fe65-690b-4ed2-886f-743571baf027" containerID="2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996" exitCode=0
Oct 29 22:24:46.964667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.964256    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"
Oct 29 22:24:46.964667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.964286    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9" event={"ID":"0151fe65-690b-4ed2-886f-743571baf027","Type":"ContainerDied","Data":"2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996"}
Oct 29 22:24:46.964667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.964351    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9" event={"ID":"0151fe65-690b-4ed2-886f-743571baf027","Type":"ContainerDied","Data":"31bd36dbd32e53816e1d98ceac5806b838917be5f46c6cec5268edf94cdaa231"}
Oct 29 22:24:46.964667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.964374    4451 scope.go:122] "RemoveContainer" containerID="2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996"
Oct 29 22:24:46.973375 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.973355    4451 scope.go:122] "RemoveContainer" containerID="2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996"
Oct 29 22:24:46.973629 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:24:46.973611    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996\": container with ID starting with 2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996 not found: ID does not exist" containerID="2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996"
Oct 29 22:24:46.973678 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:46.973639    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996"} err="failed to get container status \"2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996\": rpc error: code = NotFound desc = could not find container \"2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996\": container with ID starting with 2c707b0c7588253f6e5098afa5a3dcf9ac1e9c723697654166eba6d01d420996 not found: ID does not exist"
Oct 29 22:24:47.007265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:47.007242    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"]
Oct 29 22:24:47.017619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:47.017597    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9"]
Oct 29 22:24:48.161262 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:24:48.161229    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0151fe65-690b-4ed2-886f-743571baf027" path="/var/lib/kubelet/pods/0151fe65-690b-4ed2-886f-743571baf027/volumes"
Oct 29 22:24:48.740679 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:24:48.740638    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:25:18.740373 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:25:18.740336    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:25:41.821232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.821149    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"]
Oct 29 22:25:41.821695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.821562    4451 cpu_manager.go:414] "containerMap: removing container" podUID="0151fe65-690b-4ed2-886f-743571baf027" containerName="manager"
Oct 29 22:25:41.821695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.821574    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0151fe65-690b-4ed2-886f-743571baf027" containerName="manager"
Oct 29 22:25:41.821695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.821640    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="0151fe65-690b-4ed2-886f-743571baf027" containerName="manager"
Oct 29 22:25:41.824348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.824333    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.844049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.844022    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"]
Oct 29 22:25:41.900585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900548    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900593    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900670    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900707    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900732    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900773    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900825    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ck7\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900864    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:41.900935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:41.900884    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.001688 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001654    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.001688 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001690    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.001940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001713    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ck7\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.001940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001750    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.001940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001775    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.001940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001905    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.002216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001949    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.002216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.001973    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.002216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.002009    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.002216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.002187    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.002424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.002403    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.002503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.002480    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.003054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.003028    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.003736 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.003707    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.005189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.005165    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.006344 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.006312    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.019163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.019137    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ck7\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.019685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.019355    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-mg4hv\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.136292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.136210    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:42.454662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.454636    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"]
Oct 29 22:25:42.457117 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:25:42.457090    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477ae66c_2bff_4d2b_b0c8_724b317fe5d7.slice/crio-0be3dfa34b3c52d532f0b54cd2eede4ffe5624699d0b31dcda304f83fd702be4 WatchSource:0}: Error finding container 0be3dfa34b3c52d532f0b54cd2eede4ffe5624699d0b31dcda304f83fd702be4: Status 404 returned error can't find the container with id 0be3dfa34b3c52d532f0b54cd2eede4ffe5624699d0b31dcda304f83fd702be4
Oct 29 22:25:42.459119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.459087    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:25:42.459223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.459170    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:25:42.459223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:42.459214    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:25:43.231947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:43.231891    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" event={"ID":"477ae66c-2bff-4d2b-b0c8-724b317fe5d7","Type":"ContainerStarted","Data":"b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a"}
Oct 29 22:25:43.232366 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:43.231963    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" event={"ID":"477ae66c-2bff-4d2b-b0c8-724b317fe5d7","Type":"ContainerStarted","Data":"0be3dfa34b3c52d532f0b54cd2eede4ffe5624699d0b31dcda304f83fd702be4"}
Oct 29 22:25:43.293934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:43.290401    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podStartSLOduration=2.290383301 podStartE2EDuration="2.290383301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:43.279127093 +0000 UTC m=+683.222085811" watchObservedRunningTime="2026-10-29 22:25:43.290383301 +0000 UTC m=+683.233342007"
Oct 29 22:25:44.136401 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:44.136362    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:44.137695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:44.137673    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:25:44.137759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:44.137731    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 29 22:25:45.137196 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:45.137151    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:25:45.137681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:45.137214    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 29 22:25:46.137380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:46.137340    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:25:46.137847 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:46.137445    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 29 22:25:47.136574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:47.136527    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:25:47.136779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:47.136593    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 29 22:25:48.137559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:48.137526    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:25:48.138097 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:48.137578    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 29 22:25:48.740023 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:25:48.739977    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:25:49.137559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:49.137522    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:25:49.137765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:49.137574    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 29 22:25:50.711682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:50.711646    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:50.712246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:50.712219    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:25:50.713043 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:25:50.713021    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:26:18.739634 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:26:18.739595    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:26:41.806513 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.806480    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"]
Oct 29 22:26:41.810102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.810074    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:41.842988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.842873    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"]
Oct 29 22:26:41.880040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.880011    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-784e7a-serving-cert\""
Oct 29 22:26:41.950088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.950056    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:41.950088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.950092    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzl29\" (UniqueName: \"kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:41.950292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.950194    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:41.950292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:41.950225    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.051014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.050977    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.051014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.051015    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzl29\" (UniqueName: \"kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.051249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.051073    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.051249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.051095    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.051635 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.051614    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.053603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.053583    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.053685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.053620    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.068029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.067971    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzl29\" (UniqueName: \"kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29\") pod \"maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.124535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.124495    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:42.419445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.419414    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"]
Oct 29 22:26:42.424526 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:26:42.424496    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193bef37_cee5_4696_967d_db3ad0bd76c8.slice/crio-fb4f1106e4a61c6af78dde71ed3fec064369f5ec6f46303db8a498d5606d9fbe WatchSource:0}: Error finding container fb4f1106e4a61c6af78dde71ed3fec064369f5ec6f46303db8a498d5606d9fbe: Status 404 returned error can't find the container with id fb4f1106e4a61c6af78dde71ed3fec064369f5ec6f46303db8a498d5606d9fbe
Oct 29 22:26:42.426104 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.426083    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:26:42.492993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:42.492958    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd" event={"ID":"193bef37-cee5-4696-967d-db3ad0bd76c8","Type":"ContainerStarted","Data":"fb4f1106e4a61c6af78dde71ed3fec064369f5ec6f46303db8a498d5606d9fbe"}
Oct 29 22:26:43.498518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:43.498482    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd" event={"ID":"193bef37-cee5-4696-967d-db3ad0bd76c8","Type":"ContainerStarted","Data":"46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2"}
Oct 29 22:26:43.498905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:43.498635    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:43.534692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:43.534645    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd" podStartSLOduration=2.160065935 podStartE2EDuration="2.534631741s" totalImagesPullingTime="374.565806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:43.532827926 +0000 UTC m=+743.475786633" watchObservedRunningTime="2026-10-29 22:26:43.534631741 +0000 UTC m=+743.477590445"
Oct 29 22:26:45.832789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.832755    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"]
Oct 29 22:26:45.836582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.836562    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.866454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.866411    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"]
Oct 29 22:26:45.887150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.887107    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.887327 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.887157    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.887406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.887384    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9zw\" (UniqueName: \"kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.887500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.887474    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.921809 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.921784    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-fc684b-serving-cert\""
Oct 29 22:26:45.988330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.988289    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.988330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.988324    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.988572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.988378    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9zw\" (UniqueName: \"kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.988572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.988402    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.988572 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:26:45.988531    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-serving-cert: secret "maas-api-e2e-worker-w1-fc684b-serving-cert" not found
Oct 29 22:26:45.988729 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:26:45.988596    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls podName:6670afdc-0e6d-485e-9f49-5b8578979c1e nodeName:}" failed. No retries permitted until 2026-10-29 22:26:46.48857929 +0000 UTC m=+746.431537991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls") pod "maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" (UID: "6670afdc-0e6d-485e-9f49-5b8578979c1e") : secret "maas-api-e2e-worker-w1-fc684b-serving-cert" not found
Oct 29 22:26:45.989017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.988997    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:45.990826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:45.990805    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:46.005899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:46.005876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9zw\" (UniqueName: \"kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:46.492858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:46.492816    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:46.495262 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:46.495239    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:46.752481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:46.752397    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:47.073981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.073947    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"]
Oct 29 22:26:47.075867 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:26:47.075840    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6670afdc_0e6d_485e_9f49_5b8578979c1e.slice/crio-c4d10b6db966753fe994109eedcb3bc852682c149e5eb0e42767ba580659f412 WatchSource:0}: Error finding container c4d10b6db966753fe994109eedcb3bc852682c149e5eb0e42767ba580659f412: Status 404 returned error can't find the container with id c4d10b6db966753fe994109eedcb3bc852682c149e5eb0e42767ba580659f412
Oct 29 22:26:47.518666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.518635    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" event={"ID":"6670afdc-0e6d-485e-9f49-5b8578979c1e","Type":"ContainerStarted","Data":"c4d10b6db966753fe994109eedcb3bc852682c149e5eb0e42767ba580659f412"}
Oct 29 22:26:47.678524 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.678491    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"]
Oct 29 22:26:47.682106 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.682082    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.700431 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.700398    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2e046a-serving-cert\""
Oct 29 22:26:47.712463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.712435    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"]
Oct 29 22:26:47.806475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.806401    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.806475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.806460    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kfdb\" (UniqueName: \"kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.806648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.806556    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.806648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.806622    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.908156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.908126    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.908369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.908190    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.908369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.908235    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kfdb\" (UniqueName: \"kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.908369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.908259    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.909131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.909102    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.911597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.911574    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.912873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.912851    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.925338 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.925315    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kfdb\" (UniqueName: \"kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb\") pod \"maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:47.995502 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:47.995478    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:48.293999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:48.293970    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"]
Oct 29 22:26:48.296084 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:26:48.296041    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6d20e8_d115_4189_b8b5_219f110d80d1.slice/crio-0acb36b82ba2753168226e240a0c7ba5d7de1680beb942722c1432be5b80cb3f WatchSource:0}: Error finding container 0acb36b82ba2753168226e240a0c7ba5d7de1680beb942722c1432be5b80cb3f: Status 404 returned error can't find the container with id 0acb36b82ba2753168226e240a0c7ba5d7de1680beb942722c1432be5b80cb3f
Oct 29 22:26:48.527499 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:48.527466    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z" event={"ID":"0e6d20e8-d115-4189-b8b5-219f110d80d1","Type":"ContainerStarted","Data":"0acb36b82ba2753168226e240a0c7ba5d7de1680beb942722c1432be5b80cb3f"}
Oct 29 22:26:48.529698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:48.529663    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" event={"ID":"6670afdc-0e6d-485e-9f49-5b8578979c1e","Type":"ContainerStarted","Data":"18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba"}
Oct 29 22:26:48.529890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:48.529791    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:48.572420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:48.572240    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" podStartSLOduration=3.146466484 podStartE2EDuration="3.572219749s" totalImagesPullingTime="425.753265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:48.572108511 +0000 UTC m=+748.515067220" watchObservedRunningTime="2026-10-29 22:26:48.572219749 +0000 UTC m=+748.515178455"
Oct 29 22:26:48.740993 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:26:48.740961    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:26:49.509235 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.509207    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 22:26:49.535616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.535585    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z" event={"ID":"0e6d20e8-d115-4189-b8b5-219f110d80d1","Type":"ContainerStarted","Data":"58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192"}
Oct 29 22:26:49.535760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.535679    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:49.589297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.589221    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z" podStartSLOduration=2.246376914 podStartE2EDuration="2.589201038s" totalImagesPullingTime="342.824124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.580463911 +0000 UTC m=+749.523422604" watchObservedRunningTime="2026-10-29 22:26:49.589201038 +0000 UTC m=+749.532159744"
Oct 29 22:26:49.622150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.622116    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"]
Oct 29 22:26:49.626691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.626667    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.643967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.641904    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-22d634-serving-cert\""
Oct 29 22:26:49.644616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.644593    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"]
Oct 29 22:26:49.727624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.727589    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.727777 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.727663    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.727777 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.727741    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.727850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.727785    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzlh\" (UniqueName: \"kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.828889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.828858    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzlh\" (UniqueName: \"kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.829067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.828907    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.829067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.829025    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.829163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.829104    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.829577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.829554    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.831394 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.831371    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.831630 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.831607    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.846780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.846757    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzlh\" (UniqueName: \"kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh\") pod \"maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:49.939562 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:49.939526    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:50.242464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:50.242434    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"]
Oct 29 22:26:50.244089 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:26:50.244060    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff77874_4d39_4430_a3a4_f92e49bc33d2.slice/crio-7089aa18d1e8a282f36a2e31288916dfcfc7a0ab696907c6a94727c56b22c24d WatchSource:0}: Error finding container 7089aa18d1e8a282f36a2e31288916dfcfc7a0ab696907c6a94727c56b22c24d: Status 404 returned error can't find the container with id 7089aa18d1e8a282f36a2e31288916dfcfc7a0ab696907c6a94727c56b22c24d
Oct 29 22:26:50.541004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:50.540925    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6" event={"ID":"dff77874-4d39-4430-a3a4-f92e49bc33d2","Type":"ContainerStarted","Data":"7089aa18d1e8a282f36a2e31288916dfcfc7a0ab696907c6a94727c56b22c24d"}
Oct 29 22:26:51.547382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:51.547339    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6" event={"ID":"dff77874-4d39-4430-a3a4-f92e49bc33d2","Type":"ContainerStarted","Data":"e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658"}
Oct 29 22:26:51.547781 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:51.547437    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:26:51.583885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:51.583840    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6" podStartSLOduration=2.222049323 podStartE2EDuration="2.583827502s" totalImagesPullingTime="361.778179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:51.582992961 +0000 UTC m=+751.525951664" watchObservedRunningTime="2026-10-29 22:26:51.583827502 +0000 UTC m=+751.526786207"
Oct 29 22:26:53.306907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.306874    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"]
Oct 29 22:26:53.310950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.310927    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.335685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.335656    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2ba10a-serving-cert\""
Oct 29 22:26:53.339252 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.339224    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"]
Oct 29 22:26:53.367179 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.367156    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xcgq\" (UniqueName: \"kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.367300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.367211    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.367300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.367292    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.367400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.367317    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.467926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.467887    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.468090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.467953    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.468090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.467975    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.468090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.468049    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xcgq\" (UniqueName: \"kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.468090 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:26:53.468051    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-serving-cert: secret "maas-api-e2e-worker-w2-2ba10a-serving-cert" not found
Oct 29 22:26:53.468324 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:26:53.468123    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls podName:1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae nodeName:}" failed. No retries permitted until 2026-10-29 22:26:53.968101457 +0000 UTC m=+753.911060151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls") pod "maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" (UID: "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae") : secret "maas-api-e2e-worker-w2-2ba10a-serving-cert" not found
Oct 29 22:26:53.468795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.468776    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.470438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.470420    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.486340 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.486313    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xcgq\" (UniqueName: \"kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.973713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.973676    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:53.981974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:53.981940    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:54.224147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:54.224072    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:54.518454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:54.518422    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"]
Oct 29 22:26:54.519839 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:26:54.519808    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef5b0a3_a1ae_43cc_8a3e_c4f06eb903ae.slice/crio-36a78f4fcb26d0d1942f78fb1555442462072a81cfdb99c12422f0c5d4315f6a WatchSource:0}: Error finding container 36a78f4fcb26d0d1942f78fb1555442462072a81cfdb99c12422f0c5d4315f6a: Status 404 returned error can't find the container with id 36a78f4fcb26d0d1942f78fb1555442462072a81cfdb99c12422f0c5d4315f6a
Oct 29 22:26:54.541264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:54.541243    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 22:26:54.561433 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:54.561405    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" event={"ID":"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae","Type":"ContainerStarted","Data":"36a78f4fcb26d0d1942f78fb1555442462072a81cfdb99c12422f0c5d4315f6a"}
Oct 29 22:26:55.546288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:55.546256    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 22:26:55.568498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:55.568469    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" event={"ID":"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae","Type":"ContainerStarted","Data":"907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221"}
Oct 29 22:26:55.568641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:55.568518    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:26:55.616977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:55.616922    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" podStartSLOduration=2.371314977 podStartE2EDuration="2.616894363s" totalImagesPullingTime="245.579386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:55.614453409 +0000 UTC m=+755.557412121" watchObservedRunningTime="2026-10-29 22:26:55.616894363 +0000 UTC m=+755.559853067"
Oct 29 22:26:57.557898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:26:57.557868    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 22:27:01.579761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:27:01.579732    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 22:27:18.740860 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:27:18.740821    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:27:48.741785 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:27:48.741737    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:28:05.080004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.079970    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"]
Oct 29 22:28:05.084244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.084221    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.101067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.101040    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"]
Oct 29 22:28:05.108888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.108860    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"kube-root-ca.crt\""
Oct 29 22:28:05.111275 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.111251    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"openshift-service-ca.crt\""
Oct 29 22:28:05.111423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.111310    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"premium-w0-2e046a-kserve-self-signed-certs\""
Oct 29 22:28:05.111423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.111360    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"default-dockercfg-ht7xw\""
Oct 29 22:28:05.176387 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.176323    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx79\" (UniqueName: \"kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.176574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.176476    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.176981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.176951    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.177119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.177041    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.177119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.177089    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.177228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.177133    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.278671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.278620    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx79\" (UniqueName: \"kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.278861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.278687    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.278861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.278718    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.278861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.278793    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.278861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.278843    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.279109 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.278892    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.279206 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.279179    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.279293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.279270    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.279820 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.279797    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.282063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.281995    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.289509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.289481    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.300532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.300502    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx79\" (UniqueName: \"kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79\") pod \"premium-w0-2e046a-kserve-6dcff5674b-4q7cz\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") " pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.409607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.409531    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:05.725600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.725477    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"]
Oct 29 22:28:05.728578 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:28:05.728553    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0976adfc_ecbc_4bb8_94fa_9693dfbcef40.slice/crio-4769c7348a0159730656d3cde1bb46db792f848fd40b13af120d8c97cfca65a9 WatchSource:0}: Error finding container 4769c7348a0159730656d3cde1bb46db792f848fd40b13af120d8c97cfca65a9: Status 404 returned error can't find the container with id 4769c7348a0159730656d3cde1bb46db792f848fd40b13af120d8c97cfca65a9
Oct 29 22:28:05.926963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:05.926904    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" event={"ID":"0976adfc-ecbc-4bb8-94fa-9693dfbcef40","Type":"ContainerStarted","Data":"4769c7348a0159730656d3cde1bb46db792f848fd40b13af120d8c97cfca65a9"}
Oct 29 22:28:06.933633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:06.933593    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" event={"ID":"0976adfc-ecbc-4bb8-94fa-9693dfbcef40","Type":"ContainerStarted","Data":"fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780"}
Oct 29 22:28:06.971597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:06.971547    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podStartSLOduration=0.955874531 podStartE2EDuration="1.971532166s" totalImagesPullingTime="1.015657635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:06.96929771 +0000 UTC m=+826.912256416" watchObservedRunningTime="2026-10-29 22:28:06.971532166 +0000 UTC m=+826.914490871"
Oct 29 22:28:07.410077 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:07.410044    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:11.848346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.848310    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"]
Oct 29 22:28:11.852826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.852799    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:11.881658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.881632    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"kube-root-ca.crt\""
Oct 29 22:28:11.881817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.881781    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"premium-w5-22d634-kserve-self-signed-certs\""
Oct 29 22:28:11.881817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.881728    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"default-dockercfg-x59xw\""
Oct 29 22:28:11.882013 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.881681    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"openshift-service-ca.crt\""
Oct 29 22:28:11.883817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.883795    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"]
Oct 29 22:28:11.948400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.948367    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55rf\" (UniqueName: \"kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:11.948595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.948415    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:11.948595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.948442    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:11.948762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.948722    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:11.948824 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.948776    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:11.948882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:11.948865    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.050432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.050399    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.050634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.050452    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.050634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.050593    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.050634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.050619    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.050983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.050942    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.050983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.050971    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.051207 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.051184    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.051307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.051225    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55rf\" (UniqueName: \"kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.051508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.051488    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.054213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.054191    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.057668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.057643    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.069864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.069834    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55rf\" (UniqueName: \"kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf\") pod \"premium-w5-22d634-kserve-75d4df9f4-tgpqx\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") " pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.168037 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.167966    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:12.473189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.473161    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"]
Oct 29 22:28:12.476440 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:28:12.476415    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5656101a_f6cd_4f3f_884b_f645ea165f35.slice/crio-104905e141b8b966020b8e45fdf5ae6ab03739bcb262ce23fc44a249ae766a0c WatchSource:0}: Error finding container 104905e141b8b966020b8e45fdf5ae6ab03739bcb262ce23fc44a249ae766a0c: Status 404 returned error can't find the container with id 104905e141b8b966020b8e45fdf5ae6ab03739bcb262ce23fc44a249ae766a0c
Oct 29 22:28:12.964952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.964904    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" event={"ID":"5656101a-f6cd-4f3f-884b-f645ea165f35","Type":"ContainerStarted","Data":"df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739"}
Oct 29 22:28:12.965436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:12.964962    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" event={"ID":"5656101a-f6cd-4f3f-884b-f645ea165f35","Type":"ContainerStarted","Data":"104905e141b8b966020b8e45fdf5ae6ab03739bcb262ce23fc44a249ae766a0c"}
Oct 29 22:28:13.001098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:13.001048    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podStartSLOduration=2.001033436 podStartE2EDuration="2.001033436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:12.999342821 +0000 UTC m=+832.942301525" watchObservedRunningTime="2026-10-29 22:28:13.001033436 +0000 UTC m=+832.943992145"
Oct 29 22:28:13.168753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:13.168723    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:15.410438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:15.410400    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:15.428542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:15.428513    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:15.991186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:15.991155    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:28:18.741345 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:28:18.741313    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:28:22.168730 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:22.168703    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:22.180812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:22.180784    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:23.031035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:23.031005    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:28:48.740358 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:28:48.740263    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:28:50.717901 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.717862    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"]
Oct 29 22:28:50.723247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.723227    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.753290 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.753138    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2e046a-openshift-default-dockercfg-5g58t\""
Oct 29 22:28:50.756893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.756866    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"]
Oct 29 22:28:50.831702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.831673    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.831867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.831746    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.831867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.831840    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.831961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.831936    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jd8p\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.831999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.831970    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.832047 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.832029    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.832081 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.832062    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.832114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.832087    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.832189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.832170    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933326    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933378    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933491    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jd8p\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933608 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933533    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933667    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933694    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933722    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933751    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933781    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933762    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.933854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933839    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.934092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.933875    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.934142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.934090    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.934397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.934373    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.936041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.936017    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.936194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.936175    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.956292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.956262    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:50.960603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:50.960580    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jd8p\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:51.037346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:51.037261    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:51.361490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:51.361451    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"]
Oct 29 22:28:51.363619 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:28:51.363584    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0ef58b_50d5_42ec_a0e9_2f842d540cb0.slice/crio-a53b6a9e40780698d31f21fae0eafc2123ca8442f6de3b40d6f3cda5242d62df WatchSource:0}: Error finding container a53b6a9e40780698d31f21fae0eafc2123ca8442f6de3b40d6f3cda5242d62df: Status 404 returned error can't find the container with id a53b6a9e40780698d31f21fae0eafc2123ca8442f6de3b40d6f3cda5242d62df
Oct 29 22:28:51.366083 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:51.366043    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:28:51.366175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:51.366121    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:28:51.366240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:51.366175    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:28:52.163465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:52.163423    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" event={"ID":"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0","Type":"ContainerStarted","Data":"f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2"}
Oct 29 22:28:52.163983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:52.163480    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" event={"ID":"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0","Type":"ContainerStarted","Data":"a53b6a9e40780698d31f21fae0eafc2123ca8442f6de3b40d6f3cda5242d62df"}
Oct 29 22:28:52.215933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:52.206841    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podStartSLOduration=2.206822554 podStartE2EDuration="2.206822554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:52.204468509 +0000 UTC m=+872.147427244" watchObservedRunningTime="2026-10-29 22:28:52.206822554 +0000 UTC m=+872.149781260"
Oct 29 22:28:53.038153 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:53.038119    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:53.039794 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:53.039767    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:28:53.039953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:53.039814    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 22:28:54.038458 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:54.038420    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:28:54.038883 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:54.038471    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 22:28:55.038228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:55.038191    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:28:55.038435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:55.038253    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 22:28:56.038075 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:56.038042    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:28:56.038570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:56.038115    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 22:28:57.038603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:57.038566    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:28:57.039063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:57.038618    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 22:28:58.056968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:58.056898    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:58.057490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:58.057461    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:28:58.058362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:28:58.058338    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:29:16.587188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.587154    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"]
Oct 29 22:29:16.595355 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.595320    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608300    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmcrp\" (UniqueName: \"kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608369    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608407    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608435    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608492    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608538    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.608895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.608837    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"]
Oct 29 22:29:16.615748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.615721    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"distinct-w0-2e046a-kserve-self-signed-certs\""
Oct 29 22:29:16.709124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709094    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709139    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709157    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709202    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709234    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709304    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmcrp\" (UniqueName: \"kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709710    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709713    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.709998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.709979    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.712018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.711992    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.712412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.712393    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.726376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.726353    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmcrp\" (UniqueName: \"kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp\") pod \"distinct-w0-2e046a-kserve-794b89c4c7-gghk6\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:16.911543 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:16.911443    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:17.207894 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:17.207869    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"]
Oct 29 22:29:17.210881 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:29:17.210854    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c631028_0a42_445c_be49_b8aff3255b33.slice/crio-5622948663a28da905a432d83df2940bf858f3bc08be3d300631d5c4e3df6939 WatchSource:0}: Error finding container 5622948663a28da905a432d83df2940bf858f3bc08be3d300631d5c4e3df6939: Status 404 returned error can't find the container with id 5622948663a28da905a432d83df2940bf858f3bc08be3d300631d5c4e3df6939
Oct 29 22:29:17.290551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:17.290112    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" event={"ID":"4c631028-0a42-445c-be49-b8aff3255b33","Type":"ContainerStarted","Data":"ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6"}
Oct 29 22:29:17.290551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:17.290162    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" event={"ID":"4c631028-0a42-445c-be49-b8aff3255b33","Type":"ContainerStarted","Data":"5622948663a28da905a432d83df2940bf858f3bc08be3d300631d5c4e3df6939"}
Oct 29 22:29:17.327622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:17.327564    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podStartSLOduration=1.3275460190000001 podStartE2EDuration="1.327546019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:17.32457491 +0000 UTC m=+897.267533616" watchObservedRunningTime="2026-10-29 22:29:17.327546019 +0000 UTC m=+897.270504730"
Oct 29 22:29:17.911882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:17.911848    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:18.740295 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:29:18.740247    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:29:26.912134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:26.912097    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:26.924833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:26.924809    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:27.349354 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:27.349328    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:34.816469 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.816431    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"]
Oct 29 22:29:34.821525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.821501    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:34.834539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.834496    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"default-dockercfg-k2fzk\""
Oct 29 22:29:34.844923 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.844885    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"kube-root-ca.crt\""
Oct 29 22:29:34.845066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.844881    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"openshift-service-ca.crt\""
Oct 29 22:29:34.845066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.844995    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"distinct-w3-43b37d-kserve-self-signed-certs\""
Oct 29 22:29:34.847184 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.847159    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"]
Oct 29 22:29:34.902515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.902479    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:34.902691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.902536    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:34.902691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.902666    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:34.902808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.902789    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:34.902858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.902824    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ngf2\" (UniqueName: \"kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:34.902924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:34.902858    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004262    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004312    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ngf2\" (UniqueName: \"kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004449    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004542    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004576    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004659    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.004976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.004952    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.005248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.005218    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.005446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.005421    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.007080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.007058    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.007220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.007205    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.033664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.033590    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ngf2\" (UniqueName: \"kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2\") pod \"distinct-w3-43b37d-kserve-676c87888c-ztf48\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.056181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.056149    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"]
Oct 29 22:29:35.060362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.060338    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.074488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.074462    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-784e7a-openshift-default-dockercfg-hn6qv\""
Oct 29 22:29:35.085349 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.085322    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"]
Oct 29 22:29:35.136564 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.136528    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.208951    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209052    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209096    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209197    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209242    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209287    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209315    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kz5c\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209348    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.209378    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.213100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.210618    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"]
Oct 29 22:29:35.218526 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.218500    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.235895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.234437    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-43b37d-openshift-default-dockercfg-g742w\""
Oct 29 22:29:35.236099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.235994    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"]
Oct 29 22:29:35.310571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310542    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.310751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310593    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.310751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310617    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.310751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310710    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.310953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310790    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.310953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310857    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.310953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310889    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5l6\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.310967    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311007    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311030    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311065    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311075    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311101    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kz5c\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311129    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311186    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311131    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311260    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.311493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311414    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311427    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.311493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311458    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.311692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311533    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.311692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311600    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.312013 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.311990    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.313739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.313718    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.314288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.314267    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.328614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.328555    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.328780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.328761    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kz5c\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.334314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.334292    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"]
Oct 29 22:29:35.338929 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.338901    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.366887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.357273    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-22d634-openshift-default-dockercfg-8sxjc\""
Oct 29 22:29:35.371390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.371365    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"]
Oct 29 22:29:35.379583 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.379558    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:35.412311 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412265    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.412436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412321    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412369    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412402    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412438    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412462    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412496    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412534    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.412687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412558    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412586    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm25v\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.412687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412628    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.412687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412645    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5l6\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.412833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412760    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.412992    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413088    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413107    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413117    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413192    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413233    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413272    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413391    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413505    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.413864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.413824    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.419377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.419350    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.420853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.420830    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.441926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.441881    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5l6\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.442144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.442042    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"]
Oct 29 22:29:35.443964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.443892    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.472863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.464331    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"]
Oct 29 22:29:35.472863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.470203    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.480186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.480165    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"]
Oct 29 22:29:35.482539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.482496    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fc684b-openshift-default-dockercfg-8hgq7\""
Oct 29 22:29:35.514533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514502    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514557    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514589    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514621    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514644    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514680    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514718    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514747    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm25v\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.514896 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.514872    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.516282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.515979    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.516282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.516076    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.516282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.516247    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.516529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.516324    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.516529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.516504    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.519273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.519052    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.519442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.519422    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.531978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.531947    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.532173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.532155    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm25v\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.542184 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.542163    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:35.577624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.577587    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"]
Oct 29 22:29:35.583595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.583519    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.605148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.602147    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2ba10a-openshift-default-dockercfg-n9ldm\""
Oct 29 22:29:35.605148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.603893    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"]
Oct 29 22:29:35.616597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616566    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.616738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616660    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.616738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616700    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.616848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616770    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.616848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616799    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.616848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616821    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.617015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616957    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.617015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.616998    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.617099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.617057    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsgl\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.653114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.653089    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:35.704087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.704049    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"]
Oct 29 22:29:35.716397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.715618    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.716397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.715705    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.716397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.715748    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717700    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717745    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717775    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717811    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717843    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717871    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717952    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.719716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.717981    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720258    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720325    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720380    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720454    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720528    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720573    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720623    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rpz\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720662    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsgl\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720728    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.720761    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.721700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.721178    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.723228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.722836    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.723322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.723238    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.725502 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.724639    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.725502 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.724873    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.727117 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.726825    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.727180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.727127    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.738216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.738193    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.740904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.740856    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsgl\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.788798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.788774    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822477    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rpz\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822581    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822606    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822644    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822671    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822713    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822785    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822841    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.822939    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.823383    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.824010    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.824521    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.825173    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.825888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.825498    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.830331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.830287    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.830659 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.830635    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.841052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.839383    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"]
Oct 29 22:29:35.845552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.845059    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.846744 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.846691    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rpz\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.852321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.851941    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.872140 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.872112    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"]
Oct 29 22:29:35.874902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.874882    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"]
Oct 29 22:29:35.886015 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:29:35.885984    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9ef1bd_4959_469e_9b12_4df1f39de316.slice/crio-e6174f5f92ec9d0091b1f8f74e1c33e2ec9dc7c31cf0256add898a4de27f345d WatchSource:0}: Error finding container e6174f5f92ec9d0091b1f8f74e1c33e2ec9dc7c31cf0256add898a4de27f345d: Status 404 returned error can't find the container with id e6174f5f92ec9d0091b1f8f74e1c33e2ec9dc7c31cf0256add898a4de27f345d
Oct 29 22:29:35.890681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.890645    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.890875 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.890851    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.893634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.893198    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.913118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.912839    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925608    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925703    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925758    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925807    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnlq\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925845    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925884    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925946    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.925991    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.926237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.926045    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:35.979036 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.979005    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"]
Oct 29 22:29:35.984298 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:29:35.984152    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945fa94d_f6f9_4270_b17d_3b40c04860c4.slice/crio-b67e9ed704aee975788c4fdb18ed88c2cd64ed0bb449c37b323e0fcb31ee19c4 WatchSource:0}: Error finding container b67e9ed704aee975788c4fdb18ed88c2cd64ed0bb449c37b323e0fcb31ee19c4: Status 404 returned error can't find the container with id b67e9ed704aee975788c4fdb18ed88c2cd64ed0bb449c37b323e0fcb31ee19c4
Oct 29 22:29:35.988614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.987870    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.988614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.987961    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:35.988614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:35.988005    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.026730    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnlq\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.026792    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.026830    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.026871    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.026926    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.026981    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.027028    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.027102    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.027155    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.027540    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.027952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.027760    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.029028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.028736    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.039030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.038360    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.050475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.050428    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.051051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.051008    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.051562 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.051246    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.054391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.054365    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnlq\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.056701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.056633    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.153507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.151642    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"]
Oct 29 22:29:36.167661 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.166939    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.167661 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.167049    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.167661 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.167098    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.170374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.169560    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:36.278801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.278714    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"]
Oct 29 22:29:36.312739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.297820    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.312739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.305440    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.312739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.305549    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.424774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.419009    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" event={"ID":"945fa94d-f6f9-4270-b17d-3b40c04860c4","Type":"ContainerStarted","Data":"d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b"}
Oct 29 22:29:36.424774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.419070    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" event={"ID":"945fa94d-f6f9-4270-b17d-3b40c04860c4","Type":"ContainerStarted","Data":"b67e9ed704aee975788c4fdb18ed88c2cd64ed0bb449c37b323e0fcb31ee19c4"}
Oct 29 22:29:36.428213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.427353    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" event={"ID":"4d9ef1bd-4959-469e-9b12-4df1f39de316","Type":"ContainerStarted","Data":"5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299"}
Oct 29 22:29:36.428213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.427417    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" event={"ID":"4d9ef1bd-4959-469e-9b12-4df1f39de316","Type":"ContainerStarted","Data":"e6174f5f92ec9d0091b1f8f74e1c33e2ec9dc7c31cf0256add898a4de27f345d"}
Oct 29 22:29:36.438055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.437953    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" event={"ID":"2212fa81-5872-42bf-9acd-831276cfec3e","Type":"ContainerStarted","Data":"424334aeb7d67eba6c3b02d2de39553b57d11808e9ea8906a55a3b2f3c4b56f8"}
Oct 29 22:29:36.440964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.440932    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" event={"ID":"72ab741f-a370-4992-b61e-b3b1866fdeeb","Type":"ContainerStarted","Data":"3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8"}
Oct 29 22:29:36.442373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.442350    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" event={"ID":"72ab741f-a370-4992-b61e-b3b1866fdeeb","Type":"ContainerStarted","Data":"a1a240bb273ed2f18b8a334f49822b7d2f5bf5e1dec2210d43269f62be6c4851"}
Oct 29 22:29:36.444823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.444794    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" event={"ID":"a75d478e-e111-427c-b820-a5a33314e693","Type":"ContainerStarted","Data":"d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387"}
Oct 29 22:29:36.446680 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.446659    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" event={"ID":"a75d478e-e111-427c-b820-a5a33314e693","Type":"ContainerStarted","Data":"d9f1b28c74b94d3a20f08b727fd075a4cc5be2d3c2cfc5da93c148014d9072cd"}
Oct 29 22:29:36.448724 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.448701    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" event={"ID":"de5c8db1-3048-414d-822b-2aa15c83932b","Type":"ContainerStarted","Data":"afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6"}
Oct 29 22:29:36.449426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.449408    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" event={"ID":"de5c8db1-3048-414d-822b-2aa15c83932b","Type":"ContainerStarted","Data":"2585c3af9a584d8f6c897873f80d0658038f2443bd4cf97611fbce043b4fa3ed"}
Oct 29 22:29:36.469200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.467201    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podStartSLOduration=1.467185526 podStartE2EDuration="1.467185526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.46514328 +0000 UTC m=+916.408101987" watchObservedRunningTime="2026-10-29 22:29:36.467185526 +0000 UTC m=+916.410144233"
Oct 29 22:29:36.524945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.516251    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podStartSLOduration=1.5162065340000002 podStartE2EDuration="1.516206534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.514675976 +0000 UTC m=+916.457634682" watchObservedRunningTime="2026-10-29 22:29:36.516206534 +0000 UTC m=+916.459165239"
Oct 29 22:29:36.559218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.541706    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"]
Oct 29 22:29:36.559218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.547045    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:36.559218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.553821    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.559218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.553897    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.559218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.553964    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:29:36.584871 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.584812    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podStartSLOduration=1.584791401 podStartE2EDuration="1.584791401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.58341491 +0000 UTC m=+916.526373617" watchObservedRunningTime="2026-10-29 22:29:36.584791401 +0000 UTC m=+916.527750108"
Oct 29 22:29:36.658464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.658370    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:36.660171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.660109    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podStartSLOduration=2.660086564 podStartE2EDuration="2.660086564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.635449493 +0000 UTC m=+916.578408200" watchObservedRunningTime="2026-10-29 22:29:36.660086564 +0000 UTC m=+916.603045266"
Oct 29 22:29:36.673942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.673019    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"]
Oct 29 22:29:36.682899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.682868    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.722937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.706050    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"kube-root-ca.crt\""
Oct 29 22:29:36.784874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.713994    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"openshift-service-ca.crt\""
Oct 29 22:29:36.784874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.714128    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"distinct-w2-2ba10a-kserve-self-signed-certs\""
Oct 29 22:29:36.784874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.714254    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"default-dockercfg-56tcf\""
Oct 29 22:29:36.784874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.765378    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"]
Oct 29 22:29:36.784874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.773202    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podStartSLOduration=1.773181504 podStartE2EDuration="1.773181504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.698953351 +0000 UTC m=+916.641912067" watchObservedRunningTime="2026-10-29 22:29:36.773181504 +0000 UTC m=+916.716140207"
Oct 29 22:29:36.801981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.795007    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:36.867799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.867769    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.868445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.868422    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.868672 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.868652    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.868793 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.868778    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gdvg\" (UniqueName: \"kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.868980 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.868964    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.870507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.869106    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.973544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.972621    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.973544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.972727    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.973544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.972772    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gdvg\" (UniqueName: \"kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.973544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.972861    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.973544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.972899    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.973544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.972993    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.975829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.974577    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.975829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.974872    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.975829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.975443    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.986481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.985302    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:36.988587 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.988528    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:37.001185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:36.999348    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gdvg\" (UniqueName: \"kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg\") pod \"distinct-w2-2ba10a-kserve-6b7557d598-h99gf\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:37.064133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.060167    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:37.149977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.137026    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:37.385322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.382891    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:37.385322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.383072    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.385322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.383119    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:37.481419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.481380    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" event={"ID":"f96b660b-d2ed-4dac-8881-a25f40e8913a","Type":"ContainerStarted","Data":"878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b"}
Oct 29 22:29:37.481680 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.481661    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" event={"ID":"f96b660b-d2ed-4dac-8881-a25f40e8913a","Type":"ContainerStarted","Data":"01c3b59cc8630ecf3616fd68e642749493c2ce7e1f322a1e33ea4afec12debe5"}
Oct 29 22:29:37.521018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.499983    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" event={"ID":"2212fa81-5872-42bf-9acd-831276cfec3e","Type":"ContainerStarted","Data":"2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148"}
Oct 29 22:29:37.573057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.573020    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.573331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.573298    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:37.584944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.580630    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podStartSLOduration=2.580611493 podStartE2EDuration="2.580611493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:37.534367104 +0000 UTC m=+917.477325835" watchObservedRunningTime="2026-10-29 22:29:37.580611493 +0000 UTC m=+917.523570199"
Oct 29 22:29:37.596316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.596241    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podStartSLOduration=2.5962221149999998 podStartE2EDuration="2.596222115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:37.585755769 +0000 UTC m=+917.528714475" watchObservedRunningTime="2026-10-29 22:29:37.596222115 +0000 UTC m=+917.539180823"
Oct 29 22:29:37.723163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.723127    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.723443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.723410    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:37.836934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.816551    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.836934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.816611    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:37.937071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.936977    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:37.985379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.946316    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.985664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.985633    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:37.986035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:37.968056    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:38.056255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.054862    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:38.252884 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.206487    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:38.254208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.225039    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.254208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.254145    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:38.417345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.417299    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"]
Oct 29 22:29:38.426849 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.426032    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.431118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.431084    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:38.549461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.549424    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.556937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.549684    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:38.611936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.591503    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" event={"ID":"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f","Type":"ContainerStarted","Data":"0b95280282c57d36a735fc365f211c824a0e8ada0ce0f45ae6684b2b22122c38"}
Oct 29 22:29:38.668581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.668547    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.681943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.668789    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:38.791958 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.791441    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.791958 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.791503    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:38.920946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.917440    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.920946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:38.917503    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:39.172343 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.172303    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.172994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.172960    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:39.253175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.253117    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:39.254327 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.254293    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:29:39.364962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.356033    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:39.384935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.380501    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.384935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.380562    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:39.547712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.547151    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.547712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.547235    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:39.665956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.665292    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.665956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.665361    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:39.793948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.790298    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.793948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.790359    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:39.926939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.926440    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.926939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:39.926517    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:40.181993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.174307    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.181993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.174369    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:40.313200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.313156    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:40.380788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.380356    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.380788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.380414    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:40.542711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.542625    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.543414 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.543379    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:40.666385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.666347    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.667369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.667337    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:40.723267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.722209    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:29:40.993943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.979685    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.993943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.979757    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:40.996080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.996047    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.996262 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:40.996238    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:41.232966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.219153    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.232966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.219237    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:41.393551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.393513    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.393854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.393810    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:41.551146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.551111    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.552781 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.551377    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:41.654599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.654507    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.659806 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.659752    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:41.770298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.770260    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" event={"ID":"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f","Type":"ContainerStarted","Data":"1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09"}
Oct 29 22:29:41.803329 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.803297    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.803616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.803581    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:41.922022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.918300    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.922022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:41.918361    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:42.004648 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.004025    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:42.065936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.064996    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:42.176997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.173962    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.176997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.174020    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:42.320049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.320002    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:29:42.349042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.348994    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:42.383481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.383439    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.383778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.383728    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:42.545467 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.545374    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.545728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.545701    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:42.678016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.677970    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.678293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.678267    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:42.797175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.797087    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.797428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.797396    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:42.952959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.933407    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.982540 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:42.933473    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:43.035936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.021693    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:43.035936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.022146    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:29:43.212934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.212280    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.212934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.212348    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:43.212934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.212732    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 29 22:29:43.255943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.245802    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podStartSLOduration=7.245780404 podStartE2EDuration="7.245780404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:41.862943377 +0000 UTC m=+921.805902080" watchObservedRunningTime="2026-10-29 22:29:43.245780404 +0000 UTC m=+923.188739110"
Oct 29 22:29:43.333937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.331563    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:43.333937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.332024    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:29:43.376991 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.373265    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:43.402624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.383266    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.428934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.407147    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:43.428934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.404242    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:43.544997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.544309    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.544997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.544368    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:43.678935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.676233    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.678935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.676298    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:43.792936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.792123    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.792936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.792190    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:43.923936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.923283    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.923936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:43.923345    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:44.024936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.024231    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:44.179996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.179102    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.179996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.179162    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:44.342934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.342029    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:44.394940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.391250    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.394940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.391313    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:44.408937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.407867    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:44.546406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.546316    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.547439 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.547402    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:44.654335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.654296    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.654621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.654591    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:44.794938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.792298    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.794938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.792362    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:44.916933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.916344    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.916933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:44.916419    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:45.054056 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.034023    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:45.161937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.161039    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:45.188121 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.188024    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.188404 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.188365    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:45.346059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.346009    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:29:45.383422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.383386    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.383738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.383692    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:45.412072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.411988    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:45.542731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.542632    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.543093 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.543050    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:45.654699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.654668    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.654998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.654957    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:45.789502 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.789468    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.789794 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.789749    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:45.914339 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.914300    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.914599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:45.914565    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:46.040966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.040116    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:46.164740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.159664    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:46.208048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.208009    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:46.208306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.208265    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:46.349937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.349023    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:46.396936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.385355    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:29:46.396936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.385430    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 22:29:46.421992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.416205    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:29:46.421992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.416514    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:46.487471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.471208    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:29:46.594345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.590049    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:29:46.594345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.590123    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:29:46.737206 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.708800    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:29:46.737206 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.708856    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:29:46.864070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.864031    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:46.867933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.864316    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:46.950936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.929642    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:46.950936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:46.929736    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:47.056997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.054001    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:29:47.082933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.079970    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:47.082933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.080809    4451 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Oct 29 22:29:47.110891 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.100675    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:29:47.175938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.172515    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:47.175938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.172568    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:47.274795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.265234    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:29:47.471936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.471236    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:47.500095 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.500064    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:47.534064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.534031    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:29:47.550772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.550745    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 22:29:47.588594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.588567    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:47.618937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.605663    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:47.659280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.659238    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:29:47.814994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.805433    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:29:47.814994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.805494    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:29:47.918936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.918069    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:29:47.918936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:47.918129    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:29:48.170443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.170408    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 22:29:48.170738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.170711    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 22:29:48.353933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.351570    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:48.353933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.352548    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:48.353933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.353366    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 22:29:48.791925 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:29:48.791863    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:29:48.997166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.997136    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:48.999069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:48.999046    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:49.001446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.001422    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:29:49.063376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.063290    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:49.063670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.063652    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:49.064842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.064783    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:29:49.179243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.179205    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:49.179960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.179936    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:49.185323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:49.185302    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 22:29:51.596600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.596563    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"]
Oct 29 22:29:51.602762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.602731    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.617570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.617536    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"]
Oct 29 22:29:51.622869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.622801    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"distinct-2-w0-2e046a-kserve-self-signed-certs\""
Oct 29 22:29:51.691766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.691730    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsw7\" (UniqueName: \"kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.691959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.691790    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.691959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.691860    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.692098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.692021    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.692145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.692125    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.692251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.692231    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.794933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.794782    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.794933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.794886    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.795181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.795014    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsw7\" (UniqueName: \"kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.795181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.795053    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.795181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.795074    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.795181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.795148    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.814687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.795628    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.814687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.795883    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.814687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.796469    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.814687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.814616    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.816203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.816174    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.836944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.834683    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsw7\" (UniqueName: \"kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7\") pod \"distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") " pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:51.921549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:51.921470    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:52.222170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:52.222137    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"]
Oct 29 22:29:52.224514 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:29:52.224476    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebdd9e25_f8f5_45f4_aafe_13ac66426e22.slice/crio-147c405b4f9981f01b0ff724330f8d071e338d249670574fc7b4aee56b99c220 WatchSource:0}: Error finding container 147c405b4f9981f01b0ff724330f8d071e338d249670574fc7b4aee56b99c220: Status 404 returned error can't find the container with id 147c405b4f9981f01b0ff724330f8d071e338d249670574fc7b4aee56b99c220
Oct 29 22:29:53.135150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:53.135113    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" event={"ID":"ebdd9e25-f8f5-45f4-aafe-13ac66426e22","Type":"ContainerStarted","Data":"a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1"}
Oct 29 22:29:53.135150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:53.135154    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" event={"ID":"ebdd9e25-f8f5-45f4-aafe-13ac66426e22","Type":"ContainerStarted","Data":"147c405b4f9981f01b0ff724330f8d071e338d249670574fc7b4aee56b99c220"}
Oct 29 22:29:53.168878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:53.168834    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podStartSLOduration=2.168815241 podStartE2EDuration="2.168815241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:53.16826108 +0000 UTC m=+933.111219789" watchObservedRunningTime="2026-10-29 22:29:53.168815241 +0000 UTC m=+933.111773963"
Oct 29 22:29:53.921827 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:53.921789    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:29:57.073116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:57.073087    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:29:57.084149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:29:57.084128    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:30:01.922668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:01.922633    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:30:01.935147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:01.935128    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:30:02.200023 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:02.199942    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:30:05.836328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.836226    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"]
Oct 29 22:30:05.841147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.841122    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.859215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.859181    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"]
Oct 29 22:30:05.868155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.868120    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-08ebee-openshift-default-dockercfg-4sgmc\""
Oct 29 22:30:05.967273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967232    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967299    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967432    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967469    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmlb6\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967547    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967603    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967685    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967775 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967732    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:05.967830 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:05.967774    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068431    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068464    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmlb6\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068496    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068549    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068586    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068627    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068667    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068766    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.068950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.068836    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.069120 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.069098    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.069210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.069184    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.069256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.069234    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.069307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.069291    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.069349 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.069307    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.071027 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.071002    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.071247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.071231    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.087960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.087886    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.088274 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.088251    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmlb6\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.165038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.165003    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:06.484233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.484206    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"]
Oct 29 22:30:06.485293 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:30:06.485262    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff36cacd_fec0_41a2_aa34_d05f20689ffc.slice/crio-7111ed049a66581c8b867ca9e0f9ddc510f3a153d114706823c76f9817711e4c WatchSource:0}: Error finding container 7111ed049a66581c8b867ca9e0f9ddc510f3a153d114706823c76f9817711e4c: Status 404 returned error can't find the container with id 7111ed049a66581c8b867ca9e0f9ddc510f3a153d114706823c76f9817711e4c
Oct 29 22:30:06.487832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.487790    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:06.487945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.487879    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:06.487990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:06.487959    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:07.230663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:07.230582    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" event={"ID":"ff36cacd-fec0-41a2-aa34-d05f20689ffc","Type":"ContainerStarted","Data":"7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62"}
Oct 29 22:30:07.231232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:07.230667    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" event={"ID":"ff36cacd-fec0-41a2-aa34-d05f20689ffc","Type":"ContainerStarted","Data":"7111ed049a66581c8b867ca9e0f9ddc510f3a153d114706823c76f9817711e4c"}
Oct 29 22:30:07.274687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:07.274624    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podStartSLOduration=2.274603392 podStartE2EDuration="2.274603392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:07.274468017 +0000 UTC m=+947.217426755" watchObservedRunningTime="2026-10-29 22:30:07.274603392 +0000 UTC m=+947.217562098"
Oct 29 22:30:08.165174 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:08.165141    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:08.166765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:08.166737    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:30:08.166902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:08.166783    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:30:09.166238 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:09.166202    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:30:09.166714 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:09.166251    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:30:10.165321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:10.165283    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:30:10.165516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:10.165328    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:30:11.165855 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:11.165812    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:30:11.166351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:11.165880    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:30:12.166188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:12.166152    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:30:12.166650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:12.166205    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:30:13.169460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:13.169432    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:13.169939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:13.169688    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:13.170343 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:13.170322    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 22:30:18.860986 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:30:18.856745    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:30:36.052901 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.052860    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"]
Oct 29 22:30:36.061509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.061478    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.082576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.082545    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"]
Oct 29 22:30:36.208468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.208431    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.208713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.208693    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.208828 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.208813    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.209118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.209098    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.209255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.209241    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.209434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.209403    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764st\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.209602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.209561    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.211249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.211008    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.211249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.211167    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.312674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312570    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.312674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312633    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.312928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312730    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764st\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.312928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312758    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.312928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312789    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.312928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312876    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.313175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312945    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.313175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.312979    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.313175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.313019    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.313480 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.313451    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.314302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.314275    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.314471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.314393    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.314642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.314617    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.315116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.315095    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.316732 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.316706    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.317747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.317721    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.332247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.332222    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.332695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.332673    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-764st\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.381888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.381855    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:36.721976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.721943    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"]
Oct 29 22:30:36.723414 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:30:36.723382    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fb1c3f_d992_460d_b1ff_ee45fa257c5a.slice/crio-9ba68a507f0af4a7011253839b39e266eaa29139ac1e70d7937fe9f00da427c0 WatchSource:0}: Error finding container 9ba68a507f0af4a7011253839b39e266eaa29139ac1e70d7937fe9f00da427c0: Status 404 returned error can't find the container with id 9ba68a507f0af4a7011253839b39e266eaa29139ac1e70d7937fe9f00da427c0
Oct 29 22:30:36.725624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.725595    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:36.725689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.725656    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:36.725689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:36.725685    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:37.442801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:37.442765    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" event={"ID":"66fb1c3f-d992-460d-b1ff-ee45fa257c5a","Type":"ContainerStarted","Data":"e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68"}
Oct 29 22:30:37.443219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:37.442821    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" event={"ID":"66fb1c3f-d992-460d-b1ff-ee45fa257c5a","Type":"ContainerStarted","Data":"9ba68a507f0af4a7011253839b39e266eaa29139ac1e70d7937fe9f00da427c0"}
Oct 29 22:30:37.510934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:37.503645    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podStartSLOduration=1.503623881 podStartE2EDuration="1.503623881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:37.499259878 +0000 UTC m=+977.442218590" watchObservedRunningTime="2026-10-29 22:30:37.503623881 +0000 UTC m=+977.446582588"
Oct 29 22:30:38.382068 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:38.382040    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:38.383357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:38.383328    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:30:38.383481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:38.383377    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:30:39.382655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:39.382622    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:30:39.383086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:39.382673    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:30:40.382182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:40.382150    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:30:40.382673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:40.382204    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:30:41.382991 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:41.382957    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:30:41.383382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:41.383012    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:30:42.382936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:42.382892    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:30:42.383087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:42.382965    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:30:43.382713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:43.382681    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:30:43.382885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:43.382743    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:30:44.386821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:44.386793    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:44.387319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:44.387180    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:44.387738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:44.387718    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:30:48.740569 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:30:48.740536    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:30:50.461607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.461565    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"]
Oct 29 22:30:50.583713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.583681    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"]
Oct 29 22:30:50.584345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.584263    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.789953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.789875    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.790215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.790182    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.790401 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.790381    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.790537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.790518    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.790803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.790781    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxpd\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.795742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.795717    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.795899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.795880    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.796124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.796104    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.796284 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.796266    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.905384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.905340    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.916018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.915981    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.916420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.916395    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxpd\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.916584 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.916567    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.916720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.916704    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.916869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.916853    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.917060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.917041    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.917212 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.917195    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.917354 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.917328    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.918017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.917994    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.925986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.918322    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.926148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.918958    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.926553 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.926528    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.942449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.942417    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.977935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.975609    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxpd\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.995593 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.995534    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:50.995863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.995622    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:51.004942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:50.999492    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:51.116507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:51.116144    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:52.078876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.078756    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"]
Oct 29 22:30:52.109713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.108480    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:52.109713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.108616    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:52.109713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.108665    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:30:52.704109 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.704071    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" event={"ID":"37056749-6ab5-4d80-8025-b0e32e11601f","Type":"ContainerStarted","Data":"a7c6eaf75092ca953cc9d9a59c0835ac2a1c47daea95adc5d244cca7ee8a766c"}
Oct 29 22:30:52.704332 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.704143    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" event={"ID":"37056749-6ab5-4d80-8025-b0e32e11601f","Type":"ContainerStarted","Data":"ffb440b7aeb3092de61a0fa481403c2394053815df81fde0be6be0e3d28fa8b6"}
Oct 29 22:30:52.773175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:52.773116    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podStartSLOduration=2.773101186 podStartE2EDuration="2.773101186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:52.755228724 +0000 UTC m=+992.698187432" watchObservedRunningTime="2026-10-29 22:30:52.773101186 +0000 UTC m=+992.716059967"
Oct 29 22:30:53.121100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:53.121057    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:30:53.123287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:53.123257    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:53.123444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:53.123305    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:30:54.117066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:54.117030    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:54.117268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:54.117093    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:30:55.117078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:55.117047    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:55.117615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:55.117097    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:30:56.117515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:56.117464    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:56.117959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:56.117532    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:30:57.117571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:57.117536    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:57.117992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:57.117586    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:30:58.117476 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:58.117438    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:58.117671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:58.117497    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:30:59.116970 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:59.116939    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:30:59.117144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:30:59.116989    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 22:31:00.121187 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:00.121156    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:31:00.121688 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:00.121486    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:31:00.122249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:00.122228    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:31:05.490969 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.490902    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"]
Oct 29 22:31:05.495314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.495294    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.516051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.516019    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"]
Oct 29 22:31:05.552266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552168    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552224    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552373    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552451    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552506    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552623    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552706    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpw5v\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552771    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.552944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.552892    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.602902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.602868    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"]
Oct 29 22:31:05.607208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.607186    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.627152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.627125    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"]
Oct 29 22:31:05.653967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.653931    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654128 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.653993    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654128 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654053    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654182    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654273    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654323    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654350    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654462    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654560    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654569    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654656    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htr69\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654698    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpw5v\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654809 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654772    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654868 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654825    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654868 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654858    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.654998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654898    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.654998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.654954    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.655161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.655028    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.655161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.655058    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.655319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.655247    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.655594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.655570    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.655729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.655634    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.655729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.655659    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.657478 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.657448    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.657611 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.657591    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.675187 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.675141    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.675773 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.675740    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpw5v\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.756829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.756733    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.756849    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.756867    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.756955    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.756989    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htr69\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757018    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757054    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757087    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757132    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757303    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757414    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757576    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757616    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.757695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.757653    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.759696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.759675    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.759839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.759824    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.773413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.773388    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.773551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.773533    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htr69\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:05.815357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.815310    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:05.921632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:05.921586    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:06.124512 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.124486    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"]
Oct 29 22:31:06.126212 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:31:06.126185    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21a6e4e_b9a1_4c04_b0a6_1d60591b19bc.slice/crio-a223cfb6e95fb3bf4a2ba542b31499e0db688818283d290a2ceb7fcea5c8439e WatchSource:0}: Error finding container a223cfb6e95fb3bf4a2ba542b31499e0db688818283d290a2ceb7fcea5c8439e: Status 404 returned error can't find the container with id a223cfb6e95fb3bf4a2ba542b31499e0db688818283d290a2ceb7fcea5c8439e
Oct 29 22:31:06.128697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.128665    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:06.128776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.128730    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:06.128776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.128759    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:06.219436 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:31:06.219409    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecac9340_7c1f_4a3c_9dfb_36a0c375d1e2.slice/crio-fc4fbe6c37725ff674932ec54a1b70f6f1bdfc2f82b05868695c29fd21a47593 WatchSource:0}: Error finding container fc4fbe6c37725ff674932ec54a1b70f6f1bdfc2f82b05868695c29fd21a47593: Status 404 returned error can't find the container with id fc4fbe6c37725ff674932ec54a1b70f6f1bdfc2f82b05868695c29fd21a47593
Oct 29 22:31:06.219976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.219949    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"]
Oct 29 22:31:06.222377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.222332    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:06.222483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.222414    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:06.222483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.222452    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:06.791026 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.790980    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" event={"ID":"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc","Type":"ContainerStarted","Data":"6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5"}
Oct 29 22:31:06.791694 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.791060    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" event={"ID":"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc","Type":"ContainerStarted","Data":"a223cfb6e95fb3bf4a2ba542b31499e0db688818283d290a2ceb7fcea5c8439e"}
Oct 29 22:31:06.792566 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.792543    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" event={"ID":"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2","Type":"ContainerStarted","Data":"3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263"}
Oct 29 22:31:06.792656 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.792583    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" event={"ID":"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2","Type":"ContainerStarted","Data":"fc4fbe6c37725ff674932ec54a1b70f6f1bdfc2f82b05868695c29fd21a47593"}
Oct 29 22:31:06.815570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.815544    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:06.828202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.828151    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podStartSLOduration=1.828138917 podStartE2EDuration="1.828138917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:06.827251623 +0000 UTC m=+1006.770210329" watchObservedRunningTime="2026-10-29 22:31:06.828138917 +0000 UTC m=+1006.771097622"
Oct 29 22:31:06.867341 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.867244    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podStartSLOduration=1.867181835 podStartE2EDuration="1.867181835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:06.863585277 +0000 UTC m=+1006.806544017" watchObservedRunningTime="2026-10-29 22:31:06.867181835 +0000 UTC m=+1006.810140540"
Oct 29 22:31:06.922011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:06.921981    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:07.817972 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:07.817749    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:07.817972 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:07.817825    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:07.924126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:07.924088    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:07.924420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:07.924390    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:08.826378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:08.826337    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:08.826994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:08.826960    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:08.923370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:08.923327    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:08.923653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:08.923622    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:09.816770 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:09.816299    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:09.816770 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:09.816370    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:09.926749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:09.926311    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:09.926749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:09.926373    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:10.823936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:10.820291    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:10.823936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:10.820377    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:10.924934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:10.924290    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:10.924934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:10.924363    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:11.820361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:11.820325    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:11.821021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:11.820988    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:11.923336 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:11.923298    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:11.924652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:11.924274    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:12.816439 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:12.816397    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:12.816696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:12.816664    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:12.928397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:12.928358    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:12.929390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:12.929355    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:13.848938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:13.825471    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.848938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:13.825561    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:31:13.928169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:13.928131    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.938781 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:13.938742    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:31:15.791042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:15.772202    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:15.791042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:15.773621    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:15.791042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:15.779255    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 22:31:15.979030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:15.978991    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:16.013020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:16.012978    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.063388 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:17.063042    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:17.063388 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:17.063098    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.563422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:17.563388    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:17.565158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:17.565138    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:17.567534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:17.567513    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 22:31:18.743955 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:31:18.743904    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:31:21.088969 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.088937    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"]
Oct 29 22:31:21.094330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.094311    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.119071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.119038    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"]
Oct 29 22:31:21.221205 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221170    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.221371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221229    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.221371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221351    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.221465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221452    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.221511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221489    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.221643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221626    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.222119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221864    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.222119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221906    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.222119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.221999    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqd9z\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322703    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqd9z\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322753    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322779    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322805    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322862    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322890    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.322906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.322928    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.323265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.323003    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.323265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.323030    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.323945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.323900    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.324133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.323947    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.324246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.324071    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.324377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.324361    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.324483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.324463    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.327754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.327672    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.335182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.335160    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.339489 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.339429    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqd9z\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.339998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.339980    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.409234 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.409195    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:21.739099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.739065    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"]
Oct 29 22:31:21.740257 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:31:21.740226    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9cd01f_2fd4_4f60_927f_7db36ca56aa5.slice/crio-f8657df92a0792236ed37433bb069926a0883519c4a52570960e072ee7286f00 WatchSource:0}: Error finding container f8657df92a0792236ed37433bb069926a0883519c4a52570960e072ee7286f00: Status 404 returned error can't find the container with id f8657df92a0792236ed37433bb069926a0883519c4a52570960e072ee7286f00
Oct 29 22:31:21.743054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.743014    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:21.743136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.743099    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:21.743175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:21.743138    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:22.028376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:22.028278    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" event={"ID":"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5","Type":"ContainerStarted","Data":"fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce"}
Oct 29 22:31:22.028376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:22.028354    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" event={"ID":"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5","Type":"ContainerStarted","Data":"f8657df92a0792236ed37433bb069926a0883519c4a52570960e072ee7286f00"}
Oct 29 22:31:22.071803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:22.071740    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podStartSLOduration=1.0717195959999999 podStartE2EDuration="1.071719596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:22.066620687 +0000 UTC m=+1022.009579394" watchObservedRunningTime="2026-10-29 22:31:22.071719596 +0000 UTC m=+1022.014678301"
Oct 29 22:31:22.409651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:22.409607    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:22.411260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:22.411230    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:22.411442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:22.411280    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:23.418938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:23.416420    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:23.418938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:23.416486    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:24.411932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:24.410161    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:24.411932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:24.410245    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:25.412264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:25.412222    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:25.412872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:25.412839    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:26.437080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:26.437041    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:26.437700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:26.437664    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:27.409923 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:27.409860    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:27.410227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:27.410182    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:27.947101 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:27.947053    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.065221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.065166    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 29 22:31:28.091936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.088484    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.204118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.204021    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.221057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.220998    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.373779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.373727    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.448378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.420770    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:28.448378 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.420828    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:28.450361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:28.449670    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:31:29.228615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:29.228538    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:31:29.230926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:29.229649    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:31:29.438108 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:29.438074    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:29.438381 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:29.438349    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:29.795442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:29.795406    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:31:30.410726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:30.410685    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:31:30.411234 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:30.410769    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 22:31:31.414228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:31.414194    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:31.414726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:31.414710    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:31.415846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:31.415820    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:31:36.138074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.137994    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"]
Oct 29 22:31:36.146598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.145036    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.170664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.170397    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"]
Oct 29 22:31:36.317175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317141    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317211    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsvc\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317340    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317407    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317484    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317513    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317594    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317685    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.317760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.317724    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.418800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.418715    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsvc\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.418800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.418781    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.418836    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.418892    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.418991    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419080    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419128    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419164    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419264    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419435    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419487    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419708 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419688    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419818    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.419984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.419950    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.421780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.421760    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.422060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.422039    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.435613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.435593    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.435771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.435753    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsvc\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.479638 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.479594    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:36.806090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.806060    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"]
Oct 29 22:31:36.807273 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:31:36.807237    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9c5c4d_2fef_4281_bb15_8b8181ba9510.slice/crio-32808998ebaa0e36bc42085c01ad710f240a082ce8d84177836fd93173fbd2ba WatchSource:0}: Error finding container 32808998ebaa0e36bc42085c01ad710f240a082ce8d84177836fd93173fbd2ba: Status 404 returned error can't find the container with id 32808998ebaa0e36bc42085c01ad710f240a082ce8d84177836fd93173fbd2ba
Oct 29 22:31:36.809826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.809793    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:36.809956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.809870    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:36.809956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:36.809936    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:37.434766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:37.434730    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" event={"ID":"7c9c5c4d-2fef-4281-bb15-8b8181ba9510","Type":"ContainerStarted","Data":"d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a"}
Oct 29 22:31:37.435290 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:37.434792    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" event={"ID":"7c9c5c4d-2fef-4281-bb15-8b8181ba9510","Type":"ContainerStarted","Data":"32808998ebaa0e36bc42085c01ad710f240a082ce8d84177836fd93173fbd2ba"}
Oct 29 22:31:37.480639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:37.480604    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:37.482640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:37.482609    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:37.482792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:37.482663    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:37.484582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:37.484542    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podStartSLOduration=1.484525681 podStartE2EDuration="1.484525681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:37.482282127 +0000 UTC m=+1037.425240837" watchObservedRunningTime="2026-10-29 22:31:37.484525681 +0000 UTC m=+1037.427484442"
Oct 29 22:31:38.480590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:38.480556    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:38.481088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:38.480613    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:39.480590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:39.480559    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:39.480754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:39.480612    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:40.480666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:40.480632    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:40.480865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:40.480684    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:41.480301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:41.480269    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:41.480506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:41.480345    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:42.480558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:42.480523    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:42.480974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:42.480575    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:43.480081 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:43.480048    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:31:43.480286 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:43.480108    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 29 22:31:45.532937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:45.532002    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:45.532937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:45.532067    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.805992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:45.804990    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.092064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.092017    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.173209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.173172    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.173451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.173420    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.474997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.461105    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:31:46.554725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.553952    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.554725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.554023    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.851937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.843290    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.851937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:46.843384    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.057939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.033019    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.111993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.108134    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.126088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.126055    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:31:47.472993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.401030    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.537977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.537614    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.537977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.537686    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.561889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.560999    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.589420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.588807    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.589420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.588864    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.808449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.808343    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.983575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:47.983524    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.036400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.036356    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.038405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.038380    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:31:48.071588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.070158    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.118268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.118201    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.395961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.394325    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:31:48.404960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.403245    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.404960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.403854    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:31:48.457593 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.457563    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:31:48.587726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.587694    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:31:48.627306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.618876    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:48.627306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.618958    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.784144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.756876    4451 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:48.784855 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:48.784817    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.023052 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:31:49.022961    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:31:49.348041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:49.348011    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:49.348358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:49.348334    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:49.349173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:49.349154    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:31:50.862611 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.862576    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"]
Oct 29 22:31:50.868335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.868301    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.894025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.893996    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"]
Oct 29 22:31:50.960592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.960557    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.960592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.960595    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.960808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.960720    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.960808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.960768    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.961011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.960990    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.961059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.961031    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89rr\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.961202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.961180    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.961336 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.961220    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:50.961336 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:50.961243    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.062867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.062827    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.062876    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.062950    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.062986    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063012    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063067    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063243    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063279    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063367    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063410    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063453    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063656 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063491    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89rr\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063768    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.063790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.063780    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.065481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.065458    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.065678 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.065655    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.080143 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.080116    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.080143 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.080137    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89rr\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.182850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.182776    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:51.497856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.497823    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"]
Oct 29 22:31:51.498478 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:31:51.498448    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398e7d66_1714_44d9_9a61_0883a4cc46c6.slice/crio-f2e729160b715168bd5cca4620870396a85051bab17b74d26a1af02f6a05b965 WatchSource:0}: Error finding container f2e729160b715168bd5cca4620870396a85051bab17b74d26a1af02f6a05b965: Status 404 returned error can't find the container with id f2e729160b715168bd5cca4620870396a85051bab17b74d26a1af02f6a05b965
Oct 29 22:31:51.501355 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.501315    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:51.501442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.501413    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:51.501485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.501456    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:31:51.881791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.881687    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" event={"ID":"398e7d66-1714-44d9-9a61-0883a4cc46c6","Type":"ContainerStarted","Data":"2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775"}
Oct 29 22:31:51.881791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.881748    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" event={"ID":"398e7d66-1714-44d9-9a61-0883a4cc46c6","Type":"ContainerStarted","Data":"f2e729160b715168bd5cca4620870396a85051bab17b74d26a1af02f6a05b965"}
Oct 29 22:31:51.921531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:51.921473    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podStartSLOduration=1.9214556680000001 podStartE2EDuration="1.921455668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:51.920486042 +0000 UTC m=+1051.863444762" watchObservedRunningTime="2026-10-29 22:31:51.921455668 +0000 UTC m=+1051.864414375"
Oct 29 22:31:52.183961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:52.183876    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:52.185616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:52.185581    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:52.185753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:52.185635    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:53.183310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:53.183274    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:53.183683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:53.183329    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:54.183709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:54.183675    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:54.184169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:54.183724    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:55.183709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:55.183672    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:55.184213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:55.183741    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:56.183105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:56.183076    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:56.183301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:56.183157    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:57.183347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:57.183310    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:57.183762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:57.183361    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:58.183792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:58.183749    4451 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 29 22:31:58.184177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:58.183819    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 29 22:31:59.187834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:59.187802    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:59.188341 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:59.188211    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:31:59.188795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:31:59.188768    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:32:09.803937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:09.802153    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.413174 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:10.413126    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.581937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:10.581247    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.809984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:10.809313    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.810481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:10.810263    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:32:11.201936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.201808    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.403050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.403003    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.538991 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.429347    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.540019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.539990    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:32:11.582840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.581462    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.582840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.582253    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:32:11.849967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:11.832111    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.231992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.227457    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.433937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.418883    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:32:12.496698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.496605    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.499094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.498663    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:32:12.541248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.541208    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.557940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.557124    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.588935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.587650    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.867243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.855965    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.959801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:12.957013    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.270990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.244645    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.274335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.272985    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.400784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.400755    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:32:13.545131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.545030    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.584933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.563171    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.584933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.584063    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:32:13.605935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.599857    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.713719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.713689    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:32:13.760930 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.760888    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:32:13.779076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.778944    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:32:13.824365 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.824340    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:32:13.842309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:13.842281    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 22:32:18.741464 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:32:18.741428    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:32:28.049546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:28.049509    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"]
Oct 29 22:32:28.049960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:28.049730    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy" containerID="cri-o://b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a" gracePeriod=30
Oct 29 22:32:32.761163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.761132    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"]
Oct 29 22:32:32.777049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.777021    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.780549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.780523    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"]
Oct 29 22:32:32.791271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.791248    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"default-dockercfg-d8kph\""
Oct 29 22:32:32.800837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.800810    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"distinct-w6-784e7a-kserve-self-signed-certs\""
Oct 29 22:32:32.800837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.800810    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"openshift-service-ca.crt\""
Oct 29 22:32:32.800837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.800820    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"kube-root-ca.crt\""
Oct 29 22:32:32.870230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.870196    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.870406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.870241    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.870406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.870317    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxkt\" (UniqueName: \"kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.870406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.870385    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.870603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.870579    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.870712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.870696    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976066    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976195    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976383    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976424    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976455    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxkt\" (UniqueName: \"kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976499    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976658    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.976983    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.980510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.977426    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.985573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.985547    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:32.993962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:32.990302    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:33.018612 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.004413    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxkt\" (UniqueName: \"kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt\") pod \"distinct-w6-784e7a-kserve-88dffc699-w59mb\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:33.094974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.094930    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:33.358908 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.358883    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:32:33.383297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383267    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383305    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383353    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383441    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383469    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383528    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:33.383607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383539    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383637    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:33.383709 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383683    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383734    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383774    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token\") pod \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\" (UID: \"477ae66c-2bff-4d2b-b0c8-724b317fe5d7\") "
Oct 29 22:32:33.383943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.383830    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:33.384022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384001    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:33.384099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384078    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:33.384443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384419    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.384539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384450    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.384539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384465    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.384539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384478    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.384539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.384491    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.386150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.386120    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:33.386248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.386220    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "kube-api-access-58ck7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:33.386305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.386271    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:33.386358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.386339    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo" pod "477ae66c-2bff-4d2b-b0c8-724b317fe5d7" (UID: "477ae66c-2bff-4d2b-b0c8-724b317fe5d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:32:33.420551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.420523    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"]
Oct 29 22:32:33.421775 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:32:33.421725    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb31020f_0dc3_40f9_9f52_72462ba65bdc.slice/crio-860409a1a34a093222140a90c1f811930c67607708979a6050c93f4d66b44f00 WatchSource:0}: Error finding container 860409a1a34a093222140a90c1f811930c67607708979a6050c93f4d66b44f00: Status 404 returned error can't find the container with id 860409a1a34a093222140a90c1f811930c67607708979a6050c93f4d66b44f00
Oct 29 22:32:33.485926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.485887    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.486084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.485933    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.486084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.485949    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ck7\" (UniqueName: \"kubernetes.io/projected/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-kube-api-access-58ck7\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.486084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.485960    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477ae66c-2bff-4d2b-b0c8-724b317fe5d7-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.872986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.872946    4451 generic.go:356] "Generic (PLEG): container finished" podID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerID="b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a" exitCode=0
Oct 29 22:32:33.873665 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.873633    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" event={"ID":"477ae66c-2bff-4d2b-b0c8-724b317fe5d7","Type":"ContainerDied","Data":"b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a"}
Oct 29 22:32:33.873850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.873830    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv" event={"ID":"477ae66c-2bff-4d2b-b0c8-724b317fe5d7","Type":"ContainerDied","Data":"0be3dfa34b3c52d532f0b54cd2eede4ffe5624699d0b31dcda304f83fd702be4"}
Oct 29 22:32:33.874280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.874233    4451 scope.go:122] "RemoveContainer" containerID="b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a"
Oct 29 22:32:33.875188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.875168    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"
Oct 29 22:32:33.879360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.879330    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" event={"ID":"bb31020f-0dc3-40f9-9f52-72462ba65bdc","Type":"ContainerStarted","Data":"db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1"}
Oct 29 22:32:33.879467 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.879373    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" event={"ID":"bb31020f-0dc3-40f9-9f52-72462ba65bdc","Type":"ContainerStarted","Data":"860409a1a34a093222140a90c1f811930c67607708979a6050c93f4d66b44f00"}
Oct 29 22:32:33.906177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.906155    4451 scope.go:122] "RemoveContainer" containerID="b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a"
Oct 29 22:32:33.906704 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:32:33.906679    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a\": container with ID starting with b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a not found: ID does not exist" containerID="b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a"
Oct 29 22:32:33.906853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.906828    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a"} err="failed to get container status \"b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a\": rpc error: code = NotFound desc = could not find container \"b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a\": container with ID starting with b1fab9c5c1f8d053f4bb0a5e5b34361efa3a3fea14966623224d146455602c1a not found: ID does not exist"
Oct 29 22:32:33.988940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.979161    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" podStartSLOduration=1.979140667 podStartE2EDuration="1.979140667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:33.922421854 +0000 UTC m=+1093.865380573" watchObservedRunningTime="2026-10-29 22:32:33.979140667 +0000 UTC m=+1093.922099376"
Oct 29 22:32:33.988940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.984955    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"]
Oct 29 22:32:33.989218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:33.989120    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-mg4hv"]
Oct 29 22:32:34.096045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:34.096014    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:34.175448 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:34.175368    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" path="/var/lib/kubelet/pods/477ae66c-2bff-4d2b-b0c8-724b317fe5d7/volumes"
Oct 29 22:32:35.838938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:35.838395    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.145995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:36.145084    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.411009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:36.409991    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.776987 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:36.774524    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.790369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:36.790329    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.844934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:36.844124    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.845400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:36.844977    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:32:37.621938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:37.593844    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.650717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:37.622941    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.902048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:37.897988    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:32:38.237276 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:37.909664    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.240934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:37.965347    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.277942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:37.965413    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.541203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.504315    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:32:38.541636 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.541600    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:32:38.916134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.916074    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.917523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.917501    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:32:38.970119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.967838    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 29 22:32:38.970119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.968563    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:32:38.970119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:38.970050    4451 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.090822 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.090789    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:32:39.205449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.205366    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 22:32:39.242907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.237735    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.297738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.297400    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.423477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.423444    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 22:32:39.457990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.457126    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 22:32:39.535432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:39.535402    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 22:32:43.095763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:43.095733    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:43.109574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:43.109550    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:43.351771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:43.351696    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:32:48.742000 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:32:48.741964    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:32:52.317272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.317237    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-a20c96"]
Oct 29 22:32:52.317834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.317821    4451 cpu_manager.go:414] "containerMap: removing container" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy"
Oct 29 22:32:52.317878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.317835    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy"
Oct 29 22:32:52.317937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.317928    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="477ae66c-2bff-4d2b-b0c8-724b317fe5d7" containerName="istio-proxy"
Oct 29 22:32:52.320960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.320945    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:52.336148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.336120    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-a20c96"]
Oct 29 22:32:52.342411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.342385    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:32:52.409719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.409690    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fd5\" (UniqueName: \"kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5\") pod \"test-curl-9092-a20c96\" (UID: \"23c99167-9c10-4e70-8373-681fcf898820\") " pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:52.511586 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.511553    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fd5\" (UniqueName: \"kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5\") pod \"test-curl-9092-a20c96\" (UID: \"23c99167-9c10-4e70-8373-681fcf898820\") " pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:52.528549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.528519    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fd5\" (UniqueName: \"kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5\") pod \"test-curl-9092-a20c96\" (UID: \"23c99167-9c10-4e70-8373-681fcf898820\") " pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:52.633449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.633415    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:52.932291 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.932255    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-a20c96"]
Oct 29 22:32:52.933508 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:32:52.933463    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23c99167_9c10_4e70_8373_681fcf898820.slice/crio-2726a36d08f4acedc02b654ef6cbfdff010562e4453753b5c4d4db1a8acb9f87 WatchSource:0}: Error finding container 2726a36d08f4acedc02b654ef6cbfdff010562e4453753b5c4d4db1a8acb9f87: Status 404 returned error can't find the container with id 2726a36d08f4acedc02b654ef6cbfdff010562e4453753b5c4d4db1a8acb9f87
Oct 29 22:32:52.935683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:52.935663    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:32:53.401017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:53.400970    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-a20c96" event={"ID":"23c99167-9c10-4e70-8373-681fcf898820","Type":"ContainerStarted","Data":"2726a36d08f4acedc02b654ef6cbfdff010562e4453753b5c4d4db1a8acb9f87"}
Oct 29 22:32:54.408412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:54.408373    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-a20c96" event={"ID":"23c99167-9c10-4e70-8373-681fcf898820","Type":"ContainerStarted","Data":"da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b"}
Oct 29 22:32:54.442864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:54.442814    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-a20c96" podStartSLOduration=1.855943251 podStartE2EDuration="2.442797847s" totalImagesPullingTime="586.854596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:54.440078745 +0000 UTC m=+1114.383037451" watchObservedRunningTime="2026-10-29 22:32:54.442797847 +0000 UTC m=+1114.385756552"
Oct 29 22:32:55.255234 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:55.255203    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-a20c96"]
Oct 29 22:32:55.262380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:55.262352    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-a20c96"]
Oct 29 22:32:56.421420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:56.421370    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-a20c96" podUID="23c99167-9c10-4e70-8373-681fcf898820" containerName="test-curl-9092-a20c96" containerID="cri-o://da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b" gracePeriod=2
Oct 29 22:32:58.641810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:58.641780    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:58.700267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:58.700186    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5\" (UniqueName: \"kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5\") pod \"23c99167-9c10-4e70-8373-681fcf898820\" (UID: \"23c99167-9c10-4e70-8373-681fcf898820\") "
Oct 29 22:32:58.702699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:58.702667    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5" pod "23c99167-9c10-4e70-8373-681fcf898820" (UID: "23c99167-9c10-4e70-8373-681fcf898820"). InnerVolumeSpecName "kube-api-access-l7fd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:58.802358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:58.802318    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7fd5\" (UniqueName: \"kubernetes.io/projected/23c99167-9c10-4e70-8373-681fcf898820-kube-api-access-l7fd5\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:32:59.442863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:59.442826    4451 generic.go:356] "Generic (PLEG): container finished" podID="23c99167-9c10-4e70-8373-681fcf898820" containerID="da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b" exitCode=137
Oct 29 22:32:59.443045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:59.442886    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:32:59.443045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:59.442941    4451 scope.go:122] "RemoveContainer" containerID="da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b"
Oct 29 22:32:59.452929 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:59.452892    4451 scope.go:122] "RemoveContainer" containerID="da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b"
Oct 29 22:32:59.453246 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:32:59.453224    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b\": container with ID starting with da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b not found: ID does not exist" containerID="da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b"
Oct 29 22:32:59.453330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:32:59.453255    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b"} err="failed to get container status \"da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b\": rpc error: code = NotFound desc = could not find container \"da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b\": container with ID starting with da61e1f621740621d73d08fffa7e20f5f6c3179d82fd6c9ccf3435dc1edf401b not found: ID does not exist"
Oct 29 22:32:59.464171 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:32:59.464135    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-a20c96\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="23c99167-9c10-4e70-8373-681fcf898820" pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:33:00.204225 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:33:00.204180    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-a20c96\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="23c99167-9c10-4e70-8373-681fcf898820" pod="opendatahub/test-curl-9092-a20c96"
Oct 29 22:33:00.207404 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:00.207364    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c99167-9c10-4e70-8373-681fcf898820" path="/var/lib/kubelet/pods/23c99167-9c10-4e70-8373-681fcf898820/volumes"
Oct 29 22:33:01.205933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.203256    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.250105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.250057    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.350147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.350098    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.434209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.434166    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:33:01.463090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.462992    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.556949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.555009    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.865287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.864714    4451 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:33:01.865287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.864795    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.867163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:01.865681    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.277111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:02.277001    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.351411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:02.351348    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.366004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:02.365966    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:33:02.367516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:02.366710    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:33:02.367892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:02.367876    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 22:33:02.820940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:02.820075    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:33:05.625262 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.625212    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"]
Oct 29 22:33:05.625881 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.625866    4451 cpu_manager.go:414] "containerMap: removing container" podUID="23c99167-9c10-4e70-8373-681fcf898820" containerName="test-curl-9092-a20c96"
Oct 29 22:33:05.626003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.625881    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c99167-9c10-4e70-8373-681fcf898820" containerName="test-curl-9092-a20c96"
Oct 29 22:33:05.626003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.625997    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c99167-9c10-4e70-8373-681fcf898820" containerName="test-curl-9092-a20c96"
Oct 29 22:33:05.630945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.630893    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655000    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"]
Oct 29 22:33:05.655216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655032    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655120    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655151    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655260    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655285    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655351    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655499 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655410    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655499 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655476    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78htw\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.655562 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.655536    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756265    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756337    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756363    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756391    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756406    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756478    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756548    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756626    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78htw\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756701    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.756784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756746    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.757079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.756998    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.757079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.757036    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.757079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.757001    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.757275 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.757252    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.758900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.758879    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.759147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.759126    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.774237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.774213    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78htw\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.774337 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.774255    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:05.947507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:05.947413    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:06.128704 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.128671    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"]
Oct 29 22:33:06.134103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.134073    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162675    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwms\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162765    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162838    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162864    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162930    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162957    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.162991    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.163061    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.171141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.163087    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.178961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.175088    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"]
Oct 29 22:33:06.264655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264567    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264648    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264679    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264719    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264745    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264778    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264835    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.264888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.264859    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.265204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.265050    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwms\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.265365 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.265341    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.266568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.266537    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.267324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.267299    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.267622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.267596    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.267898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.267876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.268195 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.268153    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.276828 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.276802    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.283578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.283550    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.287437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.287412    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwms\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.288748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.288724    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"]
Oct 29 22:33:06.288748 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:33:06.288732    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad94629f_d710_4aff_8344_1843fbadffac.slice/crio-0be4a7068c0b7c4db750afff0162ba067074765d4629e367f564c4fdaf87bc2d WatchSource:0}: Error finding container 0be4a7068c0b7c4db750afff0162ba067074765d4629e367f564c4fdaf87bc2d: Status 404 returned error can't find the container with id 0be4a7068c0b7c4db750afff0162ba067074765d4629e367f564c4fdaf87bc2d
Oct 29 22:33:06.291300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.291253    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:06.291395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.291317    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:06.291395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.291347    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:06.467039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.467006    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:06.705951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.705894    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" event={"ID":"ad94629f-d710-4aff-8344-1843fbadffac","Type":"ContainerStarted","Data":"94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911"}
Oct 29 22:33:06.706385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.705979    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" event={"ID":"ad94629f-d710-4aff-8344-1843fbadffac","Type":"ContainerStarted","Data":"0be4a7068c0b7c4db750afff0162ba067074765d4629e367f564c4fdaf87bc2d"}
Oct 29 22:33:06.748396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.748349    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podStartSLOduration=1.748335188 podStartE2EDuration="1.748335188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:06.744769791 +0000 UTC m=+1126.687728497" watchObservedRunningTime="2026-10-29 22:33:06.748335188 +0000 UTC m=+1126.691293892"
Oct 29 22:33:06.801018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.800991    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"]
Oct 29 22:33:06.801653 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:33:06.801630    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6af419_5ad8_4bdc_b316_51340757cd07.slice/crio-c4aaa257b376978fb96aa70eaeb9e3b3918d28347c2902ffabe449ab94206973 WatchSource:0}: Error finding container c4aaa257b376978fb96aa70eaeb9e3b3918d28347c2902ffabe449ab94206973: Status 404 returned error can't find the container with id c4aaa257b376978fb96aa70eaeb9e3b3918d28347c2902ffabe449ab94206973
Oct 29 22:33:06.804485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.804450    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:06.804574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.804555    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:06.804621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.804605    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:06.948009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:06.947977    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:07.731934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:07.731032    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" event={"ID":"fb6af419-5ad8-4bdc-b316-51340757cd07","Type":"ContainerStarted","Data":"6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5"}
Oct 29 22:33:07.731934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:07.731102    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" event={"ID":"fb6af419-5ad8-4bdc-b316-51340757cd07","Type":"ContainerStarted","Data":"c4aaa257b376978fb96aa70eaeb9e3b3918d28347c2902ffabe449ab94206973"}
Oct 29 22:33:07.775142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:07.775078    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podStartSLOduration=1.7750580500000002 podStartE2EDuration="1.77505805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:07.771526319 +0000 UTC m=+1127.714485025" watchObservedRunningTime="2026-10-29 22:33:07.77505805 +0000 UTC m=+1127.718016757"
Oct 29 22:33:07.950395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:07.950090    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:33:07.950395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:07.950139    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:33:08.167677 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.167645    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"]
Oct 29 22:33:08.173558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.173533    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.187758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.187728    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"distinct-2-w6-784e7a-kserve-self-signed-certs\""
Oct 29 22:33:08.188540 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.188519    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"]
Oct 29 22:33:08.239973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.200868    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.239973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.201036    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.239973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.201132    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.239973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.201182    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.239973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.201248    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.239973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.201374    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fxc\" (UniqueName: \"kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.312931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.312334    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fxc\" (UniqueName: \"kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.312931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.312433    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.312931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.312555    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.312931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.312653    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.312931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.312699    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.312931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.312764    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.359938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.325639    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.359938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.334267    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.359938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.334507    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.359938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.345549    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.361441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.361405    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.371999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.371129    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fxc\" (UniqueName: \"kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc\") pod \"distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") " pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.473986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.473859    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:08.474159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.474047    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:33:08.474159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.474097    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 29 22:33:08.565057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.555200    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:08.949741 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.949305    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:33:08.949741 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:08.949365    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:33:09.047007 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.046978    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"]
Oct 29 22:33:09.048713 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:33:09.048680    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0dd0998_0241_4637_b452_e3146fd29b32.slice/crio-3d05d09acb885129076a6b1bf0426c35b8361cd8752867dde7e9197d53735170 WatchSource:0}: Error finding container 3d05d09acb885129076a6b1bf0426c35b8361cd8752867dde7e9197d53735170: Status 404 returned error can't find the container with id 3d05d09acb885129076a6b1bf0426c35b8361cd8752867dde7e9197d53735170
Oct 29 22:33:09.469117 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.469077    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:33:09.469273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.469140    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 29 22:33:09.780822 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.780742    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd" event={"ID":"e0dd0998-0241-4637-b452-e3146fd29b32","Type":"ContainerStarted","Data":"f7537e08e4c0a7d1d4c31b5c507ef7c76bd4dd606091ecd58e043a5e5e1a55c0"}
Oct 29 22:33:09.780822 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.780798    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd" event={"ID":"e0dd0998-0241-4637-b452-e3146fd29b32","Type":"ContainerStarted","Data":"3d05d09acb885129076a6b1bf0426c35b8361cd8752867dde7e9197d53735170"}
Oct 29 22:33:09.828944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.824906    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd" podStartSLOduration=1.824887666 podStartE2EDuration="1.824887666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:09.824317279 +0000 UTC m=+1129.767275985" watchObservedRunningTime="2026-10-29 22:33:09.824887666 +0000 UTC m=+1129.767846372"
Oct 29 22:33:09.949310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.949268    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:33:09.949670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:09.949627    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:33:10.467623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:10.467582    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:33:10.468296 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:10.468250    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 29 22:33:10.556129 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:10.556094    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:10.948464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:10.948428    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:33:10.948623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:10.948480    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:33:11.468937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:11.467519    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:33:11.468937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:11.467578    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 29 22:33:11.948332 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:11.948284    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:33:11.948486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:11.948348    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:33:12.469300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:12.469211    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:33:12.469300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:12.469283    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 29 22:33:12.947992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:12.947956    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:33:12.948149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:12.948009    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:33:13.467680 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:13.467643    4451 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:33:13.467893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:13.467707    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 29 22:33:14.008522 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:14.008470    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:14.009150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:14.009091    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:14.009787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:14.009764    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:33:14.471596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:14.471569    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:14.471962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:14.471940    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:14.472565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:14.472549    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:33:18.556018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:18.555979    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:18.569073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:18.569045    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:18.743303 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:33:18.743266    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:33:18.924450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:18.924419    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:33:36.802645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.802608    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"]
Oct 29 22:33:36.808308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.808280    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.824477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.824339    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:33:36.825495 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.824778    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:33:36.825495 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.825192    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rgwvk\""
Oct 29 22:33:36.825663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.825404    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:33:36.839699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.839669    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"]
Oct 29 22:33:36.878116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878084    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.878314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878262    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.878314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878299    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.878402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878320    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.878437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878408    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28k8q\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.878526 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878511    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.878561 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.878535    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.979677 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.979640    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.979864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.979685    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.979864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.979701    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.979864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.979833    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28k8q\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.980118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.980014    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.980118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.980046    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.980228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.980154    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.981014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.980985    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.982231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.982204    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.982331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.982295    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.982596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.982579    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.982782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.982762    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.997304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.997277    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:36.997445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:36.997422    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28k8q\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-267vm\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:37.121132 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:37.121105    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:37.443481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:37.443450    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"]
Oct 29 22:33:37.444644 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:33:37.444616    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46a5f45_8d76_46db_b2d6_9055f4e13071.slice/crio-d65dc734d43b0222a1ac6ca8137cae6d5664cc19fb4bfc553129b3420a4ac15f WatchSource:0}: Error finding container d65dc734d43b0222a1ac6ca8137cae6d5664cc19fb4bfc553129b3420a4ac15f: Status 404 returned error can't find the container with id d65dc734d43b0222a1ac6ca8137cae6d5664cc19fb4bfc553129b3420a4ac15f
Oct 29 22:33:38.045983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:38.045949    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" event={"ID":"d46a5f45-8d76-46db-b2d6-9055f4e13071","Type":"ContainerStarted","Data":"d65dc734d43b0222a1ac6ca8137cae6d5664cc19fb4bfc553129b3420a4ac15f"}
Oct 29 22:33:40.262521 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:40.262486    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:40.262833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:40.262554    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:33:41.070964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:41.070889    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" event={"ID":"d46a5f45-8d76-46db-b2d6-9055f4e13071","Type":"ContainerStarted","Data":"8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98"}
Oct 29 22:33:41.071289 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:41.070997    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:41.073440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:41.073411    4451 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-267vm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:33:41.073593 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:41.073464    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" podUID="d46a5f45-8d76-46db-b2d6-9055f4e13071" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:33:41.131804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:41.131738    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" podStartSLOduration=2.317041389 podStartE2EDuration="5.131716138s" totalImagesPullingTime="2.814674749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:41.119695477 +0000 UTC m=+1161.062654207" watchObservedRunningTime="2026-10-29 22:33:41.131716138 +0000 UTC m=+1161.074674844"
Oct 29 22:33:42.077836 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:33:42.077808    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:33:48.741688 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:33:48.741648    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:34:05.596760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.596723    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"]
Oct 29 22:34:05.608044 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.608024    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.626310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.626284    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"]
Oct 29 22:34:05.727059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727021    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6h7\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727070    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727197    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727257    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727339    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727382    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727499 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727487    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727562    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.727609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.727585    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.828655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828623    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.828839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828661    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.828839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828697    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.828839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828779    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828863    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828888    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828948    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6h7\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.828982    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.829071    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.829458    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.829488    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.829540    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.829581    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.829814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.829712    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.831466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.831438    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.831616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.831603    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.845788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.845766    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6h7\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.845877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.845770    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:05.926230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:05.926155    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:06.241044 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:06.241015    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"]
Oct 29 22:34:06.242121 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:34:06.242093    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfe7114_b8db_4fa6_8b35_6de9c1a95a70.slice/crio-d4015bd75c7c51ecc162fdeba68e522989988a8ccdd0c9aa383757f563216f07 WatchSource:0}: Error finding container d4015bd75c7c51ecc162fdeba68e522989988a8ccdd0c9aa383757f563216f07: Status 404 returned error can't find the container with id d4015bd75c7c51ecc162fdeba68e522989988a8ccdd0c9aa383757f563216f07
Oct 29 22:34:06.244397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:06.244366    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:34:06.244461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:06.244430    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:34:06.244495 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:06.244458    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:34:07.243548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:07.243509    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" event={"ID":"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70","Type":"ContainerStarted","Data":"c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911"}
Oct 29 22:34:07.244091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:07.243580    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" event={"ID":"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70","Type":"ContainerStarted","Data":"d4015bd75c7c51ecc162fdeba68e522989988a8ccdd0c9aa383757f563216f07"}
Oct 29 22:34:07.324872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:07.324803    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podStartSLOduration=2.324780368 podStartE2EDuration="2.324780368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:07.310385567 +0000 UTC m=+1187.253344273" watchObservedRunningTime="2026-10-29 22:34:07.324780368 +0000 UTC m=+1187.267739074"
Oct 29 22:34:07.926546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:07.926506    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:07.927960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:07.927935    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:34:07.928054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:07.927984    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:34:08.927467 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:08.927425    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:34:08.927967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:08.927493    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:34:09.927715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:09.927679    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:34:09.928268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:09.927730    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:34:10.927935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:10.927432    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:34:10.927935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:10.927491    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:34:11.928935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:11.928296    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:34:11.928935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:11.928371    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:34:12.927031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:12.926992    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:34:12.927200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:12.927058    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:34:13.978453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:13.978417    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:13.979024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:13.978896    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:13.979651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:13.979634    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:34:18.741787 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:34:18.741735    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:34:48.741321 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:34:48.741238    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:34:51.339720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.339679    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"]
Oct 29 22:34:51.344176 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.344155    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.365759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.365727    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"]
Oct 29 22:34:51.494512 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494478    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.494512 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494513    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqp8q\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.494729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494650    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.494729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494682    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.494812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494753    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.494873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494860    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.494967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494950    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.495016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.494989    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.495097 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.495082    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596460    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596552    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596580    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596620    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596668    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596684    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqp8q\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.596716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596711    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596739    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.596854    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.597037    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.597070    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.597211    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.597424    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.597621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.597602    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.598964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.598938    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.599374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.599356    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.613803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.613780    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.613899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.613873    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqp8q\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.670748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.670717    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:51.991761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.991726    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"]
Oct 29 22:34:51.992069 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:34:51.992047    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9540bf_3221_482d_8b21_8d6b535eb86c.slice/crio-0202b9e7c00b5fc566838a076b90d56465b6eb4a94fe0cfbab3f7305bca0a5f4 WatchSource:0}: Error finding container 0202b9e7c00b5fc566838a076b90d56465b6eb4a94fe0cfbab3f7305bca0a5f4: Status 404 returned error can't find the container with id 0202b9e7c00b5fc566838a076b90d56465b6eb4a94fe0cfbab3f7305bca0a5f4
Oct 29 22:34:51.994870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.994836    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:34:51.994981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.994904    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:34:51.994981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:51.994953    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:34:52.711239 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:52.711195    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" event={"ID":"cc9540bf-3221-482d-8b21-8d6b535eb86c","Type":"ContainerStarted","Data":"f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7"}
Oct 29 22:34:52.711685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:52.711283    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" event={"ID":"cc9540bf-3221-482d-8b21-8d6b535eb86c","Type":"ContainerStarted","Data":"0202b9e7c00b5fc566838a076b90d56465b6eb4a94fe0cfbab3f7305bca0a5f4"}
Oct 29 22:34:52.773615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:52.773553    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podStartSLOduration=1.773535774 podStartE2EDuration="1.773535774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:52.758704823 +0000 UTC m=+1232.701663538" watchObservedRunningTime="2026-10-29 22:34:52.773535774 +0000 UTC m=+1232.716494479"
Oct 29 22:34:53.671391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:53.671354    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:53.672885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:53.672859    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:34:53.673027 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:53.672942    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:34:54.671417 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:54.671380    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:34:54.671933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:54.671437    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:34:55.671410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:55.671376    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:34:55.672105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:55.672062    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:34:56.671403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:56.671369    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:34:56.671571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:56.671423    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:34:57.671897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:57.671852    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:34:57.672317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:57.671930    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:34:58.671639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:58.671604    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:34:58.671818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:58.671664    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:34:59.689444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:59.689412    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:59.691109 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:59.691084    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:34:59.692385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:34:59.692362    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:35:18.741816 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:35:18.741773    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:35:36.480075 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.480034    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"]
Oct 29 22:35:36.485252 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.485227    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.505413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.505386    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"]
Oct 29 22:35:36.616178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616145    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616219    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616369    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616392    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616522    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbr4d\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616615    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616708 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616692    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616745    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.616835 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.616819    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718515    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbr4d\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718595    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718646    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718726    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718868 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718775    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718877    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.718993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.718977    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.719091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.719072    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.719175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.719108    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.719460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.719437    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.719723 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.719698    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.719979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.719954    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.720324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.720302    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.723143 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.723120    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.724015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.723977    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.725043 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.725018    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.736401 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.736335    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.736752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.736691    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbr4d\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:36.809609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:36.809569    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:37.128009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:37.127980    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"]
Oct 29 22:35:37.130719 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:35:37.130691    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef458bf1_c157_4388_b37a_ef8615d1b18a.slice/crio-eca87ed7c284567ba8790a9119f9ae5f3631a889bcc6acb8e0ce87b632195bd1 WatchSource:0}: Error finding container eca87ed7c284567ba8790a9119f9ae5f3631a889bcc6acb8e0ce87b632195bd1: Status 404 returned error can't find the container with id eca87ed7c284567ba8790a9119f9ae5f3631a889bcc6acb8e0ce87b632195bd1
Oct 29 22:35:37.133516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:37.133481    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:35:37.133596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:37.133548    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:35:37.133596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:37.133581    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:35:38.128362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:38.128320    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" event={"ID":"ef458bf1-c157-4388-b37a-ef8615d1b18a","Type":"ContainerStarted","Data":"4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4"}
Oct 29 22:35:38.128807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:38.128384    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" event={"ID":"ef458bf1-c157-4388-b37a-ef8615d1b18a","Type":"ContainerStarted","Data":"eca87ed7c284567ba8790a9119f9ae5f3631a889bcc6acb8e0ce87b632195bd1"}
Oct 29 22:35:38.186067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:38.186017    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podStartSLOduration=2.186002065 podStartE2EDuration="2.186002065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:38.17404326 +0000 UTC m=+1278.117001968" watchObservedRunningTime="2026-10-29 22:35:38.186002065 +0000 UTC m=+1278.128960769"
Oct 29 22:35:38.815251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:38.815204    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:38.817479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:38.817449    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:35:38.817640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:38.817519    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 29 22:35:39.821049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:39.820995    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:35:39.821487 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:39.821072    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 29 22:35:40.811934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:40.811286    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:35:40.811934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:40.811352    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 29 22:35:41.814293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:41.814258    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:35:41.815042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:41.815000    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 29 22:35:42.811314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:42.811276    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:35:42.811614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:42.811583    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 29 22:35:43.811538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:43.811505    4451 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:35:43.812070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:43.811560    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 29 22:35:45.136243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:45.136206    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:45.137713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:45.137691    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:45.138243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:45.138224    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:35:48.741718 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:35:48.741672    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:35:50.592094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.592040    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"]
Oct 29 22:35:50.604726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.604694    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.614171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.614143    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"]
Oct 29 22:35:50.627652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.625289    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3bd0f6-serving-cert\""
Oct 29 22:35:50.758986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.758953    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.759145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.759050    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.759193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.759156    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.759228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.759214    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9gbf\" (UniqueName: \"kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.862371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.860979    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9gbf\" (UniqueName: \"kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.862371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.861213    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.862371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.861268    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.862371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.861396    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.864757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.862418    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.877497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.877466    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.877629 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.877535    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.879664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.879644    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9gbf\" (UniqueName: \"kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf\") pod \"maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:50.932716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:50.932682    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:51.244962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:51.244928    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"]
Oct 29 22:35:51.245412 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:35:51.245380    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd111483e_be3a_4478_bbfa_c55215de5da4.slice/crio-ed6b84325e74f64bf75ac1138b22b839855b067190021b558f2c55b20705f425 WatchSource:0}: Error finding container ed6b84325e74f64bf75ac1138b22b839855b067190021b558f2c55b20705f425: Status 404 returned error can't find the container with id ed6b84325e74f64bf75ac1138b22b839855b067190021b558f2c55b20705f425
Oct 29 22:35:52.224439 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:52.224398    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" event={"ID":"d111483e-be3a-4478-bbfa-c55215de5da4","Type":"ContainerStarted","Data":"880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6"}
Oct 29 22:35:52.224941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:52.224464    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" event={"ID":"d111483e-be3a-4478-bbfa-c55215de5da4","Type":"ContainerStarted","Data":"ed6b84325e74f64bf75ac1138b22b839855b067190021b558f2c55b20705f425"}
Oct 29 22:35:52.224941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:52.224528    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:52.267173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:52.267117    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" podStartSLOduration=1.919461276 podStartE2EDuration="2.267102849s" totalImagesPullingTime="347.641573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:52.262502466 +0000 UTC m=+1292.205461172" watchObservedRunningTime="2026-10-29 22:35:52.267102849 +0000 UTC m=+1292.210061553"
Oct 29 22:35:58.235489 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.235459    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 22:35:58.818097 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.818057    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"]
Oct 29 22:35:58.822204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.822183    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.840663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.840628    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"]
Oct 29 22:35:58.880028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.879993    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880078    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880143    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880238    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880264    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fl2c\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880350    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880385    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880513 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880465    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.880681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.880662    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.981795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.981763    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.981832    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.981855    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fl2c\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.981898    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.981971    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.981991    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982103    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982201    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982240    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982432    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982459    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982605    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982686 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982642    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.982686 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.982661    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.984410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.984387    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.984666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.984647    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.998762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.998737    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:58.999022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:58.999005    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fl2c\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2qx7m\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:59.142350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:59.142319    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:35:59.495746 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:59.495664    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"]
Oct 29 22:35:59.499240 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:35:59.499213    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331913b3_ed54_4d8f_866e_867d04f312b1.slice/crio-32e64d1959a3d2ba6949f94ab123da891fb34f548d1146099d40bfee620de22f WatchSource:0}: Error finding container 32e64d1959a3d2ba6949f94ab123da891fb34f548d1146099d40bfee620de22f: Status 404 returned error can't find the container with id 32e64d1959a3d2ba6949f94ab123da891fb34f548d1146099d40bfee620de22f
Oct 29 22:35:59.502795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:59.502756    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:35:59.503076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:59.502933    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:35:59.503076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:35:59.502992    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:36:00.285679 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:00.285639    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" event={"ID":"331913b3-ed54-4d8f-866e-867d04f312b1","Type":"ContainerStarted","Data":"149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685"}
Oct 29 22:36:00.285855 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:00.285716    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" event={"ID":"331913b3-ed54-4d8f-866e-867d04f312b1","Type":"ContainerStarted","Data":"32e64d1959a3d2ba6949f94ab123da891fb34f548d1146099d40bfee620de22f"}
Oct 29 22:36:01.143162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:01.143132    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:36:01.144758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:01.144734    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:01.144888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:01.144781    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:02.143302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:02.143270    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:02.143683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:02.143329    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:03.143018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:03.142937    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:03.143018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:03.142993    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:04.143497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:04.143462    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:04.143877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:04.143524    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:05.143026 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:05.142987    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:05.143225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:05.143055    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:06.143351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:06.143309    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:06.143858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:06.143381    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:07.143249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.143215    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:07.143449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.143266    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:07.607525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.607467    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podStartSLOduration=9.60745105 podStartE2EDuration="9.60745105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:00.343782308 +0000 UTC m=+1300.286741035" watchObservedRunningTime="2026-10-29 22:36:07.60745105 +0000 UTC m=+1307.550409755"
Oct 29 22:36:07.608581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.608557    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"]
Oct 29 22:36:07.613387 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.613367    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.632496 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.632455    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"]
Oct 29 22:36:07.635672 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.635648    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1fa4b8b6-serving-cert\""
Oct 29 22:36:07.676793 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.676752    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.677096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.677072    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.677244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.677196    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.677244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.677234    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcrl\" (UniqueName: \"kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.778586 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.778547    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.778792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.778609    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.778792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.778634    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcrl\" (UniqueName: \"kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.778792 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:36:07.778699    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-serving-cert: secret "maas-api-e2e-mt-1fa4b8b6-serving-cert" not found
Oct 29 22:36:07.778792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.778733    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.778792 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:36:07.778763    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls podName:f26f217e-d8a5-46e3-b007-42cfb6d12108 nodeName:}" failed. No retries permitted until 2026-10-29 22:36:08.278747233 +0000 UTC m=+1308.221705930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls") pod "maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" (UID: "f26f217e-d8a5-46e3-b007-42cfb6d12108") : secret "maas-api-e2e-mt-1fa4b8b6-serving-cert" not found
Oct 29 22:36:07.779314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.779293    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.781226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.781197    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:07.796605 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:07.796584    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcrl\" (UniqueName: \"kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:08.143087 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:08.143052    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:36:08.143288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:08.143111    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:36:08.286531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:08.286482    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:08.289213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:08.289183    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:08.535988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:08.535198    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:08.903810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:08.903780    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"]
Oct 29 22:36:08.904926 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:36:08.904876    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26f217e_d8a5_46e3_b007_42cfb6d12108.slice/crio-0150306adc0f688ec45d4965043cb9f6c5688d403ad6b69b05bb0b46a8e69eca WatchSource:0}: Error finding container 0150306adc0f688ec45d4965043cb9f6c5688d403ad6b69b05bb0b46a8e69eca: Status 404 returned error can't find the container with id 0150306adc0f688ec45d4965043cb9f6c5688d403ad6b69b05bb0b46a8e69eca
Oct 29 22:36:09.268770 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.268356    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:36:09.269419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.269398    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:36:09.270895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.270874    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:36:09.417027 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.416991    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" event={"ID":"f26f217e-d8a5-46e3-b007-42cfb6d12108","Type":"ContainerStarted","Data":"b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a"}
Oct 29 22:36:09.417599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.417578    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:09.417727 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.417712    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" event={"ID":"f26f217e-d8a5-46e3-b007-42cfb6d12108","Type":"ContainerStarted","Data":"0150306adc0f688ec45d4965043cb9f6c5688d403ad6b69b05bb0b46a8e69eca"}
Oct 29 22:36:09.455482 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:09.455432    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" podStartSLOduration=2.076700687 podStartE2EDuration="2.455417421s" totalImagesPullingTime="378.716734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:09.45175371 +0000 UTC m=+1309.394712414" watchObservedRunningTime="2026-10-29 22:36:09.455417421 +0000 UTC m=+1309.398376127"
Oct 29 22:36:16.442047 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:16.442012    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:36:18.741954 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:36:18.741877    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:36:25.787743 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.787708    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"]
Oct 29 22:36:25.792654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.792629    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.817190 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.817149    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"]
Oct 29 22:36:25.823826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.823802    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3bd0f6-openshift-default-dockercfg-cmm6g\""
Oct 29 22:36:25.894265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894231    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894281    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894392    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894579 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894557    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894628    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894669    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894784    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894818    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.894967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.894947    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvk2c\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.996707 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.996674    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.996898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.996725    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.996898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.996783    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvk2c\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.996898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.996865    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.996925    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997009    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997162    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997224    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997237    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997263    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997471    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997510    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997667    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:25.997778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:25.997759    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:26.000125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.000090    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:26.000231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.000135    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:26.017309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.017284    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:26.017546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.017528    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvk2c\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:26.117000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.116970    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:26.434895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.434592    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"]
Oct 29 22:36:26.435744 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:36:26.435708    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8724b2c4_0f22_4d90_8b0e_e6d9527a06be.slice/crio-b0a0a0dba350798943b077117e5850b81406f7d8d932615c22ad4e8ef0173f5e WatchSource:0}: Error finding container b0a0a0dba350798943b077117e5850b81406f7d8d932615c22ad4e8ef0173f5e: Status 404 returned error can't find the container with id b0a0a0dba350798943b077117e5850b81406f7d8d932615c22ad4e8ef0173f5e
Oct 29 22:36:26.440362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.440328    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:36:26.440443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.440395    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:36:26.440443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.440424    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:36:26.550547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.550511    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" event={"ID":"8724b2c4-0f22-4d90-8b0e-e6d9527a06be","Type":"ContainerStarted","Data":"a5a38e0b5872326eb4ee0bf0cb8cd0a9f54da13f06bd3c6d9dc95f589fcf90c6"}
Oct 29 22:36:26.550680 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.550568    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" event={"ID":"8724b2c4-0f22-4d90-8b0e-e6d9527a06be","Type":"ContainerStarted","Data":"b0a0a0dba350798943b077117e5850b81406f7d8d932615c22ad4e8ef0173f5e"}
Oct 29 22:36:26.589626 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:26.589573    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podStartSLOduration=1.5895586339999999 podStartE2EDuration="1.589558634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:26.585801429 +0000 UTC m=+1326.528760137" watchObservedRunningTime="2026-10-29 22:36:26.589558634 +0000 UTC m=+1326.532517340"
Oct 29 22:36:27.117375 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:27.117338    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:27.119006 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:27.118976    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:36:27.119156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:27.119033    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:36:28.117635 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:28.117599    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:36:28.118148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:28.117664    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:36:29.117552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:29.117516    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:36:29.117748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:29.117566    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:36:30.118324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:30.118283    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:36:30.118837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:30.118353    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:36:31.117383 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:31.117342    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:36:31.117597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:31.117403    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:36:32.117435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:32.117399    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:36:32.117840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:32.117453    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:36:33.124784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:33.124753    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:33.125255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:33.125194    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:33.126038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:36:33.126017    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:36:48.770701 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:36:48.770660    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:37:02.194318 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.194286    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"]
Oct 29 22:37:02.200214 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.200194    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.217114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.217058    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"]
Oct 29 22:37:02.221843 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.221816    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cae55b-serving-cert\""
Oct 29 22:37:02.349658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.349623    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.349851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.349687    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.349997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.349974    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.350072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.350037    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssxn\" (UniqueName: \"kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.451189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.451104    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.451346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.451261    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sssxn\" (UniqueName: \"kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.451409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.451395    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.451456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.451448    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.452149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.452115    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.454048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.454021    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.454139 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.454052    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.469789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.469765    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssxn\" (UniqueName: \"kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn\") pod \"maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.516105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.516075    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:02.833793 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.833766    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"]
Oct 29 22:37:02.834110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:37:02.834087    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c84a37_a8f6_4bcb_9015_227f0ea3f2ba.slice/crio-1c11be596b3e456548ef4b0cc5a3554abc5338a8a09d05bef999d3d948a9e35d WatchSource:0}: Error finding container 1c11be596b3e456548ef4b0cc5a3554abc5338a8a09d05bef999d3d948a9e35d: Status 404 returned error can't find the container with id 1c11be596b3e456548ef4b0cc5a3554abc5338a8a09d05bef999d3d948a9e35d
Oct 29 22:37:02.861777 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:02.861747    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc" event={"ID":"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba","Type":"ContainerStarted","Data":"1c11be596b3e456548ef4b0cc5a3554abc5338a8a09d05bef999d3d948a9e35d"}
Oct 29 22:37:03.870546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:03.870507    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc" event={"ID":"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba","Type":"ContainerStarted","Data":"c11a05df08790a069c51b4a87ee88e5c5c3777e696989f5e3011dcdb14944053"}
Oct 29 22:37:03.870981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:03.870596    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:03.910490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:03.910437    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc" podStartSLOduration=1.589633683 podStartE2EDuration="1.910421969s" totalImagesPullingTime="320.788286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:03.908364125 +0000 UTC m=+1363.851322830" watchObservedRunningTime="2026-10-29 22:37:03.910421969 +0000 UTC m=+1363.853380673"
Oct 29 22:37:09.882215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:09.882186    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:37:10.507516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.507480    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"]
Oct 29 22:37:10.511515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.511496    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.531145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.531075    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-616da7-serving-cert\""
Oct 29 22:37:10.543204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.543177    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"]
Oct 29 22:37:10.586018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.585979    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.586293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.586262    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.586422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.586402    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zsgn\" (UniqueName: \"kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.586556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.586536    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.688408 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.688374    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.688554 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.688513    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.688609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.688575    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zsgn\" (UniqueName: \"kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.688646 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.688638    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.689192 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.689162    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.691484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.691459    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.691598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.691459    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.705904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.705838    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zsgn\" (UniqueName: \"kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn\") pod \"maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:10.826176 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:10.826141    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:11.134907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:11.134868    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"]
Oct 29 22:37:11.135329 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:37:11.135098    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b07267_dc4e_41f3_87bc_71f81b459074.slice/crio-25dcfaeffe3ea14448d781ffdf87da832c085c6847019bff6279543f20971a21 WatchSource:0}: Error finding container 25dcfaeffe3ea14448d781ffdf87da832c085c6847019bff6279543f20971a21: Status 404 returned error can't find the container with id 25dcfaeffe3ea14448d781ffdf87da832c085c6847019bff6279543f20971a21
Oct 29 22:37:11.933831 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:11.933795    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq" event={"ID":"d4b07267-dc4e-41f3-87bc-71f81b459074","Type":"ContainerStarted","Data":"d3508c450cd4c3f5279ede7861c8960508a236ac839891829edf6f2dedbc693b"}
Oct 29 22:37:11.934058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:11.933853    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq" event={"ID":"d4b07267-dc4e-41f3-87bc-71f81b459074","Type":"ContainerStarted","Data":"25dcfaeffe3ea14448d781ffdf87da832c085c6847019bff6279543f20971a21"}
Oct 29 22:37:11.934058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:11.933921    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:11.971079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:11.971023    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq" podStartSLOduration=1.699928135 podStartE2EDuration="1.971006392s" totalImagesPullingTime="271.078257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:11.968220071 +0000 UTC m=+1371.911178778" watchObservedRunningTime="2026-10-29 22:37:11.971006392 +0000 UTC m=+1371.913965097"
Oct 29 22:37:14.011024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.010995    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"]
Oct 29 22:37:14.015479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.015452    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.076657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.076619    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"]
Oct 29 22:37:14.145585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.145550    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.145764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.145611    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.145764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.145740    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.145838 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.145799    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.145951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.145938    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.146059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.146043    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.146094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.146082    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvkd8\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.146127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.146112    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.146158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.146141    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247439 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247399    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247528    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247627    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247671    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvkd8\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247710    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247751 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247735    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247821    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.247924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247874    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.248040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.247987    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.248156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.248068    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.248156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.248128    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.248351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.248322    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.248532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.248363    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.249305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.249183    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.250887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.250861    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.251144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.251122    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.266802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.266750    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.266946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.266896    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvkd8\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hrms9\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.337892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.337868    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:14.652899 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:37:14.652867    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885ace3b_2569_476a_96b2_a17db6689b6b.slice/crio-3202a8912e2a0ef1b2fe2d031766b8bad1a4d4267303633849163537d700caeb WatchSource:0}: Error finding container 3202a8912e2a0ef1b2fe2d031766b8bad1a4d4267303633849163537d700caeb: Status 404 returned error can't find the container with id 3202a8912e2a0ef1b2fe2d031766b8bad1a4d4267303633849163537d700caeb
Oct 29 22:37:14.653143 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.653119    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"]
Oct 29 22:37:14.655626 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.655591    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:14.655732 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.655671    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:14.655732 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.655718    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:14.955841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.955759    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" event={"ID":"885ace3b-2569-476a-96b2-a17db6689b6b","Type":"ContainerStarted","Data":"00bd12c883a94f7a94429152b2d6032efeb9dbc6e4a6a43c3c5ad0dc01d8a6f6"}
Oct 29 22:37:14.955841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.955808    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" event={"ID":"885ace3b-2569-476a-96b2-a17db6689b6b","Type":"ContainerStarted","Data":"3202a8912e2a0ef1b2fe2d031766b8bad1a4d4267303633849163537d700caeb"}
Oct 29 22:37:14.996296 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:14.996229    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podStartSLOduration=1.9962100619999998 podStartE2EDuration="1.996210062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:14.994008406 +0000 UTC m=+1374.936967108" watchObservedRunningTime="2026-10-29 22:37:14.996210062 +0000 UTC m=+1374.939168769"
Oct 29 22:37:15.339376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:15.339337    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:15.340842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:15.340807    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:15.341025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:15.340874    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:16.339204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:16.339166    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:16.339403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:16.339225    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:17.338404 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.338364    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:17.338622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.338443    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:17.339939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.339894    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"]
Oct 29 22:37:17.344534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.344518    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.362396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.362374    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"]
Oct 29 22:37:17.363654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.363632    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ee886f-serving-cert\""
Oct 29 22:37:17.427251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.427194    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.427508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.427488    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqtfj\" (UniqueName: \"kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.427637 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.427624    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.427905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.427891    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.529711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.529678    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.529878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.529739    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqtfj\" (UniqueName: \"kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.529878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.529770    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.529978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.529887    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.530799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.530769    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.532789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.532764    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.532886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.532764    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.546873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.546847    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqtfj\" (UniqueName: \"kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj\") pod \"maas-api-e2e-api-a-ee886f-8595d84686-4hsm7\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.666311 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.664738    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:17.947594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:17.947508    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:37:18.030682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:18.030655    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"]
Oct 29 22:37:18.339365 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:18.339321    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:18.339569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:18.339399    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:18.768556 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:37:18.768388    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:37:18.984885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:18.984851    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7" event={"ID":"c8bf64e4-1b6c-4b49-870e-4166393e1268","Type":"ContainerStarted","Data":"a8a62371536b157fd73370e80ec37bc7eae15e36ac6c70b39a19d62cd42c82dd"}
Oct 29 22:37:18.985083 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:18.984929    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:18.985083 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:18.984956    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7" event={"ID":"c8bf64e4-1b6c-4b49-870e-4166393e1268","Type":"ContainerStarted","Data":"b12ce20487ef8e2c8e5863a44788b9de26c73e20d7cccf7e0cbdd7cfe602b5c2"}
Oct 29 22:37:19.030477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:19.030389    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7" podStartSLOduration=1.7052162549999998 podStartE2EDuration="2.030373803s" totalImagesPullingTime="325.157548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:19.028148816 +0000 UTC m=+1378.971107523" watchObservedRunningTime="2026-10-29 22:37:19.030373803 +0000 UTC m=+1378.973332507"
Oct 29 22:37:19.339279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:19.339245    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:19.339447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:19.339306    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:20.338489 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:20.338457    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:20.338974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:20.338505    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:21.339063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:21.339021    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hrms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:37:21.339595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:21.339072    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:37:23.164510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:23.164480    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:23.165064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:23.164799    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:23.165803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:23.165781    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:37:25.014278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:25.014248    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:37:48.583003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.582906    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"]
Oct 29 22:37:48.589777 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.589752    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.614825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.614798    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"]
Oct 29 22:37:48.693460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.693426    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cae55b-openshift-default-dockercfg-pxrq8\""
Oct 29 22:37:48.698291 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698269    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698312    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698467 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698453    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698487    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698527    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698690    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698807    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698870    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pq8\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.698987 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.698971    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.743032 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:37:48.742989    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:37:48.800706 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.800669    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.800888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.800756    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.800888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.800784    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.800888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.800854    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.800888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.800884    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.800974    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801042    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801171    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801231    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801282    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801348    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pq8\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801350    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.801626 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.801602    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.802073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.802046    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.803497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.803475    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.803761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.803733    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.818136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.818113    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.818444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.818422    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pq8\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:48.914335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:48.914254    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:49.247146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.247093    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"]
Oct 29 22:37:49.248674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.248629    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:49.248779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.248731    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:49.248842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.248775    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:49.669407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.669308    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" event={"ID":"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7","Type":"ContainerStarted","Data":"acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf"}
Oct 29 22:37:49.669811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.669412    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" event={"ID":"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7","Type":"ContainerStarted","Data":"23f49d0c4b209a371b042ca2c9ec6a6edee4facf3f497b55d98ddee88c5835d8"}
Oct 29 22:37:49.729755 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.729688    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podStartSLOduration=1.729672988 podStartE2EDuration="1.729672988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:49.718245308 +0000 UTC m=+1409.661204015" watchObservedRunningTime="2026-10-29 22:37:49.729672988 +0000 UTC m=+1409.672631692"
Oct 29 22:37:49.915053 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:49.915019    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:50.922839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:50.922801    4451 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:37:50.923369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:50.922868    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:37:51.914763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:51.914733    4451 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:37:51.915062 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:51.915021    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:37:52.916013 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:52.915977    4451 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:37:52.916633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:52.916596    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:37:53.917936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:53.917419    4451 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:37:53.917936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:53.917519    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:37:54.092103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.092062    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"]
Oct 29 22:37:54.118946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.113243    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"]
Oct 29 22:37:54.119380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.119265    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.136504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.135479    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1131a29d-openshift-default-dockercfg-8jjv9\""
Oct 29 22:37:54.252992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.247032    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.279105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.279068    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.279398 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.279377    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.279599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.279583    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rms8\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.279715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.279703    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.279806 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.279793    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.279928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.279898    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.280071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.280055    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.280223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.280209    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381418    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381473    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381578    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rms8\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381611    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381639    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381668    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381726    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381835    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.381874    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.382086    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.382352    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.382440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.382423    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.383039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.383018    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.383165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.383057    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.385434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.385418    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.385700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.385685    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.411080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.405723    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rms8\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.411080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.409033    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.479616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.479579    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:54.847115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.847080    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"]
Oct 29 22:37:54.856530 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:37:54.856479    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c64fdae_c173_4742_856d_1de92bce7bfa.slice/crio-e54ec77230089b2b53004c431b692185332d088b4d165e3d393c3f7912318d57 WatchSource:0}: Error finding container e54ec77230089b2b53004c431b692185332d088b4d165e3d393c3f7912318d57: Status 404 returned error can't find the container with id e54ec77230089b2b53004c431b692185332d088b4d165e3d393c3f7912318d57
Oct 29 22:37:54.859832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.859734    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:54.860077 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.860050    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:54.860226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.860208    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:37:54.939200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.936330    4451 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:37:54.939200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:54.936387    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:37:55.894625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:55.894581    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" event={"ID":"3c64fdae-c173-4742-856d-1de92bce7bfa","Type":"ContainerStarted","Data":"9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3"}
Oct 29 22:37:55.894948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:55.894906    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" event={"ID":"3c64fdae-c173-4742-856d-1de92bce7bfa","Type":"ContainerStarted","Data":"e54ec77230089b2b53004c431b692185332d088b4d165e3d393c3f7912318d57"}
Oct 29 22:37:55.915028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:55.914993    4451 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:37:55.915298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:55.915266    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:37:56.038161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.038100    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podStartSLOduration=2.038079186 podStartE2EDuration="2.038079186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:55.986371029 +0000 UTC m=+1415.929329735" watchObservedRunningTime="2026-10-29 22:37:56.038079186 +0000 UTC m=+1415.981037894"
Oct 29 22:37:56.480867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.480799    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:37:56.482905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.482877    4451 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:37:56.483048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.482954    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:37:56.928658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.928630    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:56.929187 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.929159    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:56.929954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:56.929900    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:37:57.480982 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:57.480941    4451 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:37:57.481394 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:57.481009    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:37:58.481125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:58.481093    4451 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:37:58.481515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:58.481143    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:37:59.481289 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:59.481255    4451 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:37:59.481711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:37:59.481325    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:38:00.483984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:00.482277    4451 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:38:00.483984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:00.482352    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:38:01.577397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:01.577357    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:38:01.577947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:01.577791    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:38:01.578697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:01.578674    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:38:15.478885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:15.478842    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"]
Oct 29 22:38:15.479375 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:15.479180    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" containerID="cri-o://9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3" gracePeriod=30
Oct 29 22:38:16.579562 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:16.579518    4451 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:38:16.579969 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:16.579587    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:38:18.742344 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:38:18.742305    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:38:20.766131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.766064    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:38:20.855375 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855335    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855398    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855531    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855566    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855648    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855674    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.855791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855743    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:20.855926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855896    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855905    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.855996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855956    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.855996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.855980    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.856153 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.856091    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token\") pod \"3c64fdae-c173-4742-856d-1de92bce7bfa\" (UID: \"3c64fdae-c173-4742-856d-1de92bce7bfa\") "
Oct 29 22:38:20.856208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.856169    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.856334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.856257    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.856988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.856963    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.857161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.857145    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.857267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.857255    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c64fdae-c173-4742-856d-1de92bce7bfa-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.857358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.857346    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.857451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.857438    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.858843 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.858817    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:20.858999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.858965    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "kube-api-access-5rms8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:20.859377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.859350    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:20.868713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.868675    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy" pod "3c64fdae-c173-4742-856d-1de92bce7bfa" (UID: "3c64fdae-c173-4742-856d-1de92bce7bfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.959185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.959149    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.959334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.959188    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rms8\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-kube-api-access-5rms8\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.959334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.959204    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.959334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:20.959218    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c64fdae-c173-4742-856d-1de92bce7bfa-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.219179 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.219049    4451 generic.go:356] "Generic (PLEG): container finished" podID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerID="9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3" exitCode=0
Oct 29 22:38:21.219536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.219516    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"
Oct 29 22:38:21.220013 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.219973    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" event={"ID":"3c64fdae-c173-4742-856d-1de92bce7bfa","Type":"ContainerDied","Data":"9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3"}
Oct 29 22:38:21.220340 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.220303    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk" event={"ID":"3c64fdae-c173-4742-856d-1de92bce7bfa","Type":"ContainerDied","Data":"e54ec77230089b2b53004c431b692185332d088b4d165e3d393c3f7912318d57"}
Oct 29 22:38:21.220546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.220531    4451 scope.go:122] "RemoveContainer" containerID="9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3"
Oct 29 22:38:21.254244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.254201    4451 scope.go:122] "RemoveContainer" containerID="9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3"
Oct 29 22:38:21.255473 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:38:21.255411    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3\": container with ID starting with 9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3 not found: ID does not exist" containerID="9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3"
Oct 29 22:38:21.255595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.255490    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3"} err="failed to get container status \"9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3\": rpc error: code = NotFound desc = could not find container \"9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3\": container with ID starting with 9cb3db3025f061e418a422de8f12311bff1797245ebfbeb46e2e20f17fa73dc3 not found: ID does not exist"
Oct 29 22:38:21.284148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.284105    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"]
Oct 29 22:38:21.302111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:21.302081    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-d4sqk"]
Oct 29 22:38:22.164649 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:22.164615    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" path="/var/lib/kubelet/pods/3c64fdae-c173-4742-856d-1de92bce7bfa/volumes"
Oct 29 22:38:31.555654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:31.555615    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"]
Oct 29 22:38:31.556223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:31.555867    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" podUID="f26f217e-d8a5-46e3-b007-42cfb6d12108" containerName="maas-api" containerID="cri-o://b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a" gracePeriod=30
Oct 29 22:38:32.080076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.079721    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:38:32.235444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.235410    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl\" (UniqueName: \"kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl\") pod \"f26f217e-d8a5-46e3-b007-42cfb6d12108\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") "
Oct 29 22:38:32.235653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.235464    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca\") pod \"f26f217e-d8a5-46e3-b007-42cfb6d12108\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") "
Oct 29 22:38:32.235653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.235547    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\") pod \"f26f217e-d8a5-46e3-b007-42cfb6d12108\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") "
Oct 29 22:38:32.235653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.235641    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs\") pod \"f26f217e-d8a5-46e3-b007-42cfb6d12108\" (UID: \"f26f217e-d8a5-46e3-b007-42cfb6d12108\") "
Oct 29 22:38:32.237694 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.237662    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca" pod "f26f217e-d8a5-46e3-b007-42cfb6d12108" (UID: "f26f217e-d8a5-46e3-b007-42cfb6d12108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:32.240081 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.240053    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs" pod "f26f217e-d8a5-46e3-b007-42cfb6d12108" (UID: "f26f217e-d8a5-46e3-b007-42cfb6d12108"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:32.240182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.240126    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl" pod "f26f217e-d8a5-46e3-b007-42cfb6d12108" (UID: "f26f217e-d8a5-46e3-b007-42cfb6d12108"). InnerVolumeSpecName "kube-api-access-6lcrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:32.240241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.240067    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls" pod "f26f217e-d8a5-46e3-b007-42cfb6d12108" (UID: "f26f217e-d8a5-46e3-b007-42cfb6d12108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:32.330804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.330766    4451 generic.go:356] "Generic (PLEG): container finished" podID="f26f217e-d8a5-46e3-b007-42cfb6d12108" containerID="b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a" exitCode=0
Oct 29 22:38:32.331021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.330830    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" event={"ID":"f26f217e-d8a5-46e3-b007-42cfb6d12108","Type":"ContainerDied","Data":"b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a"}
Oct 29 22:38:32.331021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.330882    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4" event={"ID":"f26f217e-d8a5-46e3-b007-42cfb6d12108","Type":"ContainerDied","Data":"0150306adc0f688ec45d4965043cb9f6c5688d403ad6b69b05bb0b46a8e69eca"}
Oct 29 22:38:32.331021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.330923    4451 scope.go:122] "RemoveContainer" containerID="b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a"
Oct 29 22:38:32.331218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.331024    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"
Oct 29 22:38:32.338147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.338119    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lcrl\" (UniqueName: \"kubernetes.io/projected/f26f217e-d8a5-46e3-b007-42cfb6d12108-kube-api-access-6lcrl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:32.338261 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.338150    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f26f217e-d8a5-46e3-b007-42cfb6d12108-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:32.338261 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.338166    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:32.338261 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.338178    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f26f217e-d8a5-46e3-b007-42cfb6d12108-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:38:32.343267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.343246    4451 scope.go:122] "RemoveContainer" containerID="b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a"
Oct 29 22:38:32.343553 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:38:32.343534    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a\": container with ID starting with b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a not found: ID does not exist" containerID="b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a"
Oct 29 22:38:32.343613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.343563    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a"} err="failed to get container status \"b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a\": rpc error: code = NotFound desc = could not find container \"b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a\": container with ID starting with b371f528ea02532f2128ed4e056047efd0491d1aec5d721d2a8f2fc165b98b2a not found: ID does not exist"
Oct 29 22:38:32.390935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.383114    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"]
Oct 29 22:38:32.390935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:32.389594    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-6cgz4"]
Oct 29 22:38:34.169161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:38:34.169117    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26f217e-d8a5-46e3-b007-42cfb6d12108" path="/var/lib/kubelet/pods/f26f217e-d8a5-46e3-b007-42cfb6d12108/volumes"
Oct 29 22:38:48.742158 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:38:48.742114    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:39:18.742644 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:39:18.742599    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:39:48.746646 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:39:48.746605    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:39:52.516807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:52.516774    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"]
Oct 29 22:39:52.517310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:52.517023    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc" podUID="c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" containerName="maas-api" containerID="cri-o://c11a05df08790a069c51b4a87ee88e5c5c3777e696989f5e3011dcdb14944053" gracePeriod=30
Oct 29 22:39:53.352229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.352131    4451 generic.go:356] "Generic (PLEG): container finished" podID="c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" containerID="c11a05df08790a069c51b4a87ee88e5c5c3777e696989f5e3011dcdb14944053" exitCode=0
Oct 29 22:39:53.352406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.352287    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc" event={"ID":"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba","Type":"ContainerDied","Data":"c11a05df08790a069c51b4a87ee88e5c5c3777e696989f5e3011dcdb14944053"}
Oct 29 22:39:53.439135 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.439105    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:39:53.507942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.507198    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn\" (UniqueName: \"kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn\") pod \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") "
Oct 29 22:39:53.507942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.507260    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs\") pod \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") "
Oct 29 22:39:53.507942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.507385    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls\") pod \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") "
Oct 29 22:39:53.507942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.507557    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca\") pod \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\" (UID: \"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba\") "
Oct 29 22:39:53.512932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.509708    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca" pod "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" (UID: "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:53.522996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.517024    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls" pod "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" (UID: "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:39:53.522996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.518533    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs" pod "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" (UID: "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:39:53.522996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.520125    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn" pod "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" (UID: "c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba"). InnerVolumeSpecName "kube-api-access-sssxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:53.610034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.609904    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sssxn\" (UniqueName: \"kubernetes.io/projected/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-kube-api-access-sssxn\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:39:53.610034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.609973    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:39:53.610034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.609985    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:39:53.610034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:53.609995    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:39:54.365285 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:54.363857    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc" event={"ID":"c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba","Type":"ContainerDied","Data":"1c11be596b3e456548ef4b0cc5a3554abc5338a8a09d05bef999d3d948a9e35d"}
Oct 29 22:39:54.365285 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:54.363992    4451 scope.go:122] "RemoveContainer" containerID="c11a05df08790a069c51b4a87ee88e5c5c3777e696989f5e3011dcdb14944053"
Oct 29 22:39:54.365285 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:54.364226    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"
Oct 29 22:39:54.409551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:54.409517    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"]
Oct 29 22:39:54.419374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:54.419346    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae55b-55cddb887c-4fwsc"]
Oct 29 22:39:56.164439 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:56.164401    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" path="/var/lib/kubelet/pods/c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba/volumes"
Oct 29 22:39:57.500784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.500752    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"]
Oct 29 22:39:57.501428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501413    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f26f217e-d8a5-46e3-b007-42cfb6d12108" containerName="maas-api"
Oct 29 22:39:57.501475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501429    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26f217e-d8a5-46e3-b007-42cfb6d12108" containerName="maas-api"
Oct 29 22:39:57.501475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501457    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" containerName="maas-api"
Oct 29 22:39:57.501475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501464    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" containerName="maas-api"
Oct 29 22:39:57.501564 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501500    4451 cpu_manager.go:414] "containerMap: removing container" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy"
Oct 29 22:39:57.501564 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501507    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy"
Oct 29 22:39:57.501627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501595    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c64fdae-c173-4742-856d-1de92bce7bfa" containerName="istio-proxy"
Oct 29 22:39:57.501627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501606    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26f217e-d8a5-46e3-b007-42cfb6d12108" containerName="maas-api"
Oct 29 22:39:57.501627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.501616    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c84a37-a8f6-4bcb-9015-227f0ea3f2ba" containerName="maas-api"
Oct 29 22:39:57.506450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.506430    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.538401 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.538364    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"]
Oct 29 22:39:57.610246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.610218    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1fa4b8b6-openshift-default-dockercfg-mnpx6\""
Oct 29 22:39:57.682533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682496    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.682702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682542    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.682702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682562    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2bvp\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.682784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682718    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.682784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682754    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.682969 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682944    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.683022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.682996    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.683057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.683048    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.683194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.683177    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784406    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784444    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784499    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784615    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784665    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784702    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784727    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2bvp\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784923 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784817    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.784923 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.784859    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.785155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.785129    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.785310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.785288    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.785392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.785320    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.785392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.785334    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.785392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.785336    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.786986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.786950    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.787266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.787245    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.801826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.801807    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.801976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.801956    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2bvp\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:57.823897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:57.823873    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:58.141523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.141495    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"]
Oct 29 22:39:58.142346 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:39:58.142318    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073a784f_7318_46b7_bead_566b6fa028ec.slice/crio-ae7913f8dcd7cf08c3138978962f062cfa7fda641c08874202544dccbb6abe93 WatchSource:0}: Error finding container ae7913f8dcd7cf08c3138978962f062cfa7fda641c08874202544dccbb6abe93: Status 404 returned error can't find the container with id ae7913f8dcd7cf08c3138978962f062cfa7fda641c08874202544dccbb6abe93
Oct 29 22:39:58.145322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.145285    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:39:58.145434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.145367    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:39:58.145434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.145413    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:39:58.392182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.392091    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" event={"ID":"073a784f-7318-46b7-bead-566b6fa028ec","Type":"ContainerStarted","Data":"4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f"}
Oct 29 22:39:58.392182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.392155    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" event={"ID":"073a784f-7318-46b7-bead-566b6fa028ec","Type":"ContainerStarted","Data":"ae7913f8dcd7cf08c3138978962f062cfa7fda641c08874202544dccbb6abe93"}
Oct 29 22:39:58.437333 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.437265    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podStartSLOduration=1.437245546 podStartE2EDuration="1.437245546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:58.436744754 +0000 UTC m=+1538.379703463" watchObservedRunningTime="2026-10-29 22:39:58.437245546 +0000 UTC m=+1538.380204252"
Oct 29 22:39:58.824223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:58.824193    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:39:59.826027 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:59.825992    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:39:59.826432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:39:59.826055    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:40:00.125997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:00.125898    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"]
Oct 29 22:40:00.126189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:00.126148    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq" podUID="d4b07267-dc4e-41f3-87bc-71f81b459074" containerName="maas-api" containerID="cri-o://d3508c450cd4c3f5279ede7861c8960508a236ac839891829edf6f2dedbc693b" gracePeriod=30
Oct 29 22:40:00.868939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:00.847279    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:40:00.868939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:00.847349    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:40:00.986935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:00.982639    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"]
Oct 29 22:40:00.986935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:00.982942    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7" podUID="c8bf64e4-1b6c-4b49-870e-4166393e1268" containerName="maas-api" containerID="cri-o://a8a62371536b157fd73370e80ec37bc7eae15e36ac6c70b39a19d62cd42c82dd" gracePeriod=30
Oct 29 22:40:01.052110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.044671    4451 generic.go:356] "Generic (PLEG): container finished" podID="d4b07267-dc4e-41f3-87bc-71f81b459074" containerID="d3508c450cd4c3f5279ede7861c8960508a236ac839891829edf6f2dedbc693b" exitCode=0
Oct 29 22:40:01.052110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.045832    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq" event={"ID":"d4b07267-dc4e-41f3-87bc-71f81b459074","Type":"ContainerDied","Data":"d3508c450cd4c3f5279ede7861c8960508a236ac839891829edf6f2dedbc693b"}
Oct 29 22:40:01.225196 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.225166    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"]
Oct 29 22:40:01.225704 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.225665    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy" containerID="cri-o://acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf" gracePeriod=30
Oct 29 22:40:01.433986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.429315    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:01.480345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.480266    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:40:01.621936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.617495    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls\") pod \"d4b07267-dc4e-41f3-87bc-71f81b459074\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") "
Oct 29 22:40:01.621936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.617612    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca\") pod \"d4b07267-dc4e-41f3-87bc-71f81b459074\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") "
Oct 29 22:40:01.621936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.617665    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs\") pod \"d4b07267-dc4e-41f3-87bc-71f81b459074\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") "
Oct 29 22:40:01.621936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.617698    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn\" (UniqueName: \"kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn\") pod \"d4b07267-dc4e-41f3-87bc-71f81b459074\" (UID: \"d4b07267-dc4e-41f3-87bc-71f81b459074\") "
Oct 29 22:40:01.621936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.621025    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca" pod "d4b07267-dc4e-41f3-87bc-71f81b459074" (UID: "d4b07267-dc4e-41f3-87bc-71f81b459074"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:01.708005 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.700055    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs" pod "d4b07267-dc4e-41f3-87bc-71f81b459074" (UID: "d4b07267-dc4e-41f3-87bc-71f81b459074"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:01.708005 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.700377    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls" pod "d4b07267-dc4e-41f3-87bc-71f81b459074" (UID: "d4b07267-dc4e-41f3-87bc-71f81b459074"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:01.728113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.723173    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b07267-dc4e-41f3-87bc-71f81b459074-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:01.728113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.723205    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:01.728113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.723219    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b07267-dc4e-41f3-87bc-71f81b459074-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:01.742001 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.732161    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:01.757057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.757027    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn" pod "d4b07267-dc4e-41f3-87bc-71f81b459074" (UID: "d4b07267-dc4e-41f3-87bc-71f81b459074"). InnerVolumeSpecName "kube-api-access-8zsgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:01.826169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.825605    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zsgn\" (UniqueName: \"kubernetes.io/projected/d4b07267-dc4e-41f3-87bc-71f81b459074-kube-api-access-8zsgn\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:01.826169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.825968    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:40:01.826169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:01.826019    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:40:02.085248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.083799    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq" event={"ID":"d4b07267-dc4e-41f3-87bc-71f81b459074","Type":"ContainerDied","Data":"25dcfaeffe3ea14448d781ffdf87da832c085c6847019bff6279543f20971a21"}
Oct 29 22:40:02.085248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.083873    4451 scope.go:122] "RemoveContainer" containerID="d3508c450cd4c3f5279ede7861c8960508a236ac839891829edf6f2dedbc693b"
Oct 29 22:40:02.085248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.084043    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"
Oct 29 22:40:02.114932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.086549    4451 generic.go:356] "Generic (PLEG): container finished" podID="c8bf64e4-1b6c-4b49-870e-4166393e1268" containerID="a8a62371536b157fd73370e80ec37bc7eae15e36ac6c70b39a19d62cd42c82dd" exitCode=0
Oct 29 22:40:02.114932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.086674    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7" event={"ID":"c8bf64e4-1b6c-4b49-870e-4166393e1268","Type":"ContainerDied","Data":"a8a62371536b157fd73370e80ec37bc7eae15e36ac6c70b39a19d62cd42c82dd"}
Oct 29 22:40:02.187935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.186868    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:40:02.241938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.236038    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"]
Oct 29 22:40:02.241938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.240677    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616da7-94bdbfb6c-lb5jq"]
Oct 29 22:40:02.430033 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.429935    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca\") pod \"c8bf64e4-1b6c-4b49-870e-4166393e1268\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") "
Oct 29 22:40:02.430033 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.429984    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls\") pod \"c8bf64e4-1b6c-4b49-870e-4166393e1268\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") "
Oct 29 22:40:02.430271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.430110    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs\") pod \"c8bf64e4-1b6c-4b49-870e-4166393e1268\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") "
Oct 29 22:40:02.430271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.430162    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj\" (UniqueName: \"kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj\") pod \"c8bf64e4-1b6c-4b49-870e-4166393e1268\" (UID: \"c8bf64e4-1b6c-4b49-870e-4166393e1268\") "
Oct 29 22:40:02.430385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.430307    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca" pod "c8bf64e4-1b6c-4b49-870e-4166393e1268" (UID: "c8bf64e4-1b6c-4b49-870e-4166393e1268"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:02.430836 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.430815    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bf64e4-1b6c-4b49-870e-4166393e1268-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:02.432268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.432235    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs" pod "c8bf64e4-1b6c-4b49-870e-4166393e1268" (UID: "c8bf64e4-1b6c-4b49-870e-4166393e1268"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:02.432692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.432669    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls" pod "c8bf64e4-1b6c-4b49-870e-4166393e1268" (UID: "c8bf64e4-1b6c-4b49-870e-4166393e1268"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:02.432788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.432717    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj" pod "c8bf64e4-1b6c-4b49-870e-4166393e1268" (UID: "c8bf64e4-1b6c-4b49-870e-4166393e1268"). InnerVolumeSpecName "kube-api-access-mqtfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:02.532927 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.532863    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:02.532927 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.532898    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8bf64e4-1b6c-4b49-870e-4166393e1268-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:02.532927 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.532923    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqtfj\" (UniqueName: \"kubernetes.io/projected/c8bf64e4-1b6c-4b49-870e-4166393e1268-kube-api-access-mqtfj\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:02.829832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.829793    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:40:02.830506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:02.829854    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:40:03.112988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.111902    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7" event={"ID":"c8bf64e4-1b6c-4b49-870e-4166393e1268","Type":"ContainerDied","Data":"b12ce20487ef8e2c8e5863a44788b9de26c73e20d7cccf7e0cbdd7cfe602b5c2"}
Oct 29 22:40:03.112988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.112034    4451 scope.go:122] "RemoveContainer" containerID="a8a62371536b157fd73370e80ec37bc7eae15e36ac6c70b39a19d62cd42c82dd"
Oct 29 22:40:03.112988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.112185    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"
Oct 29 22:40:03.161605 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.161572    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"]
Oct 29 22:40:03.171370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.171344    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee886f-8595d84686-4hsm7"]
Oct 29 22:40:03.825242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.825209    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:40:03.825414 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:03.825264    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:40:04.164074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:04.163985    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bf64e4-1b6c-4b49-870e-4166393e1268" path="/var/lib/kubelet/pods/c8bf64e4-1b6c-4b49-870e-4166393e1268/volumes"
Oct 29 22:40:04.164946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:04.164890    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b07267-dc4e-41f3-87bc-71f81b459074" path="/var/lib/kubelet/pods/d4b07267-dc4e-41f3-87bc-71f81b459074/volumes"
Oct 29 22:40:04.824295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:04.824246    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:40:04.824491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:04.824319    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:40:06.247850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.247820    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:40:06.248292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.248243    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:40:06.249107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.249084    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:40:06.515351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.515293    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:40:06.615050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615015    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615120    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615191    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615255    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615284    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.615353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615309    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615338    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615396    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.615549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615489    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615557    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.615650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615620    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615745 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615699    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert\") pod \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\" (UID: \"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7\") "
Oct 29 22:40:06.615906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.615881    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.616248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.616226    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:06.616443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.616428    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.616548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.616446    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.616548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.616460    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.616548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.616472    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.616548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.616484    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.617747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.617722    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:06.618010 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.617987    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "kube-api-access-p7pq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:06.618119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.618030    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.618119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.618065    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token" pod "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" (UID: "d4872ee3-d023-4e33-8dea-c0adb1fdcdd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:06.718384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.718354    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.718384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.718382    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.718568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.718391    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7pq8\" (UniqueName: \"kubernetes.io/projected/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-kube-api-access-p7pq8\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.718568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.718402    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.786956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.786838    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"]
Oct 29 22:40:06.787112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:06.787042    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy" containerID="cri-o://94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911" gracePeriod=30
Oct 29 22:40:07.160992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.160957    4451 generic.go:356] "Generic (PLEG): container finished" podID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerID="acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf" exitCode=0
Oct 29 22:40:07.161181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.161014    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"
Oct 29 22:40:07.161181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.161033    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" event={"ID":"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7","Type":"ContainerDied","Data":"acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf"}
Oct 29 22:40:07.161181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.161082    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9" event={"ID":"d4872ee3-d023-4e33-8dea-c0adb1fdcdd7","Type":"ContainerDied","Data":"23f49d0c4b209a371b042ca2c9ec6a6edee4facf3f497b55d98ddee88c5835d8"}
Oct 29 22:40:07.161181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.161103    4451 scope.go:122] "RemoveContainer" containerID="acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf"
Oct 29 22:40:07.172153 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.172129    4451 scope.go:122] "RemoveContainer" containerID="acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf"
Oct 29 22:40:07.172425 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:40:07.172405    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf\": container with ID starting with acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf not found: ID does not exist" containerID="acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf"
Oct 29 22:40:07.172472 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.172433    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf"} err="failed to get container status \"acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf\": rpc error: code = NotFound desc = could not find container \"acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf\": container with ID starting with acffe9df1e5f4f10114b95ddc5b1bdb3ac91d60bd133286de13ccb64da8ac8bf not found: ID does not exist"
Oct 29 22:40:07.208012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.207978    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"]
Oct 29 22:40:07.211950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:07.211922    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-lv8h9"]
Oct 29 22:40:08.166979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:08.166863    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" path="/var/lib/kubelet/pods/d4872ee3-d023-4e33-8dea-c0adb1fdcdd7/volumes"
Oct 29 22:40:12.062317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.062257    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:40:12.126640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.126606    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.126829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.126706    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.126829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.126776    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.126861    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.126925    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127023    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127106    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127239 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127208    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127239 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127203    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.127348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127259    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo\") pod \"ad94629f-d710-4aff-8344-1843fbadffac\" (UID: \"ad94629f-d710-4aff-8344-1843fbadffac\") "
Oct 29 22:40:12.127348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127284    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.127454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127386    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.127663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.127640    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:12.128123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.128095    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.128219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.128124    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.128219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.128140    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.128219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.128154    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad94629f-d710-4aff-8344-1843fbadffac-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.128481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.128459    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.130220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.130184    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.130879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.130853    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:12.131006 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.130886    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "kube-api-access-78htw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:12.131742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.131717    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token" pod "ad94629f-d710-4aff-8344-1843fbadffac" (UID: "ad94629f-d710-4aff-8344-1843fbadffac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:12.202433 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.202397    4451 generic.go:356] "Generic (PLEG): container finished" podID="ad94629f-d710-4aff-8344-1843fbadffac" containerID="94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911" exitCode=0
Oct 29 22:40:12.202636 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.202471    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"
Oct 29 22:40:12.202636 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.202484    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" event={"ID":"ad94629f-d710-4aff-8344-1843fbadffac","Type":"ContainerDied","Data":"94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911"}
Oct 29 22:40:12.202764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.202640    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc" event={"ID":"ad94629f-d710-4aff-8344-1843fbadffac","Type":"ContainerDied","Data":"0be4a7068c0b7c4db750afff0162ba067074765d4629e367f564c4fdaf87bc2d"}
Oct 29 22:40:12.202764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.202710    4451 scope.go:122] "RemoveContainer" containerID="94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911"
Oct 29 22:40:12.215623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.215591    4451 scope.go:122] "RemoveContainer" containerID="94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911"
Oct 29 22:40:12.215895 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:40:12.215865    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911\": container with ID starting with 94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911 not found: ID does not exist" containerID="94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911"
Oct 29 22:40:12.216094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.215907    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911"} err="failed to get container status \"94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911\": rpc error: code = NotFound desc = could not find container \"94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911\": container with ID starting with 94f13154a4f97971c9543597cb22087ab9998d51dd77f22090f772c10c1c4911 not found: ID does not exist"
Oct 29 22:40:12.229923 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.229885    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad94629f-d710-4aff-8344-1843fbadffac-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.230119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.230090    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78htw\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-kube-api-access-78htw\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.230119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.230116    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.230227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.230131    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad94629f-d710-4aff-8344-1843fbadffac-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.230227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.230144    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad94629f-d710-4aff-8344-1843fbadffac-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.250840 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.250807    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"]
Oct 29 22:40:12.254465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:12.254445    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-r7bbc"]
Oct 29 22:40:14.165799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:14.165754    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad94629f-d710-4aff-8344-1843fbadffac" path="/var/lib/kubelet/pods/ad94629f-d710-4aff-8344-1843fbadffac/volumes"
Oct 29 22:40:18.743187 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:40:18.743146    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:40:27.588057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.588025    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"]
Oct 29 22:40:27.588887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.588865    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.588888    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.588907    4451 cpu_manager.go:414] "containerMap: removing container" podUID="d4b07267-dc4e-41f3-87bc-71f81b459074" containerName="maas-api"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.588931    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b07267-dc4e-41f3-87bc-71f81b459074" containerName="maas-api"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589000    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c8bf64e4-1b6c-4b49-870e-4166393e1268" containerName="maas-api"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589008    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bf64e4-1b6c-4b49-870e-4166393e1268" containerName="maas-api"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589023    4451 cpu_manager.go:414] "containerMap: removing container" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy"
Oct 29 22:40:27.589045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589030    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy"
Oct 29 22:40:27.589392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589197    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bf64e4-1b6c-4b49-870e-4166393e1268" containerName="maas-api"
Oct 29 22:40:27.589392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589215    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4872ee3-d023-4e33-8dea-c0adb1fdcdd7" containerName="istio-proxy"
Oct 29 22:40:27.589392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589229    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad94629f-d710-4aff-8344-1843fbadffac" containerName="istio-proxy"
Oct 29 22:40:27.589392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.589239    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b07267-dc4e-41f3-87bc-71f81b459074" containerName="maas-api"
Oct 29 22:40:27.594894 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.594872    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.617000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.616973    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"]
Oct 29 22:40:27.759153 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759110    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759218    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759250    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvbd\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759281    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759309    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759412    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759558    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759586    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.759610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.759601    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861031    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861080    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvbd\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861121    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861160    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861228    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861434    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861470    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861494    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861539    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861571    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861748    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861766    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.861926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861846    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.862024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.861996    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.864619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.864590    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.864748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.864731    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.878811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.878567    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.878811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.878766    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvbd\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:27.923202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:27.923167    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:28.243572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.243542    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"]
Oct 29 22:40:28.243968 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:40:28.243937    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36c9256_6d81_4b63_bb9d_9e9b5016f19a.slice/crio-a859575726aec5ae74eb9b5119ede0fa5869802eafde6fc29bf944746d664be6 WatchSource:0}: Error finding container a859575726aec5ae74eb9b5119ede0fa5869802eafde6fc29bf944746d664be6: Status 404 returned error can't find the container with id a859575726aec5ae74eb9b5119ede0fa5869802eafde6fc29bf944746d664be6
Oct 29 22:40:28.247018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.246987    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:40:28.247120 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.247049    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:40:28.247120 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.247084    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:40:28.319266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.319232    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" event={"ID":"c36c9256-6d81-4b63-bb9d-9e9b5016f19a","Type":"ContainerStarted","Data":"a859575726aec5ae74eb9b5119ede0fa5869802eafde6fc29bf944746d664be6"}
Oct 29 22:40:28.358144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.358082    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podStartSLOduration=1.358063074 podStartE2EDuration="1.358063074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:28.35614676 +0000 UTC m=+1568.299105488" watchObservedRunningTime="2026-10-29 22:40:28.358063074 +0000 UTC m=+1568.301021779"
Oct 29 22:40:28.926044 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:28.926003    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:29.328876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:29.328841    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" event={"ID":"c36c9256-6d81-4b63-bb9d-9e9b5016f19a","Type":"ContainerStarted","Data":"ae58b824fd258918e431ace6e2bc1ecc9a8562b1900baa2bcdebf0a86c920cea"}
Oct 29 22:40:29.924844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:29.924808    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:40:29.925059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:29.924858    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:40:30.923553 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:30.923518    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:40:30.924088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:30.923573    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:40:31.925030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:31.924156    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:40:31.925030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:31.924226    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:40:32.924219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:32.924179    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:40:32.924546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:32.924511    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:40:33.926255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:33.926197    4451 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:40:33.926653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:33.926291    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:40:35.528142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:35.528101    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:35.529071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:35.529044    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:35.530361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:35.530340    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:40:37.453706 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:37.453669    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"]
Oct 29 22:40:37.454237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:37.453863    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy" containerID="cri-o://2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775" gracePeriod=30
Oct 29 22:40:42.742744 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.742715    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:40:42.772162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772130    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772188    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772223    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772254    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772342    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772379    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772420    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.772607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772455    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772516    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772632    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo\") pod \"398e7d66-1714-44d9-9a61-0883a4cc46c6\" (UID: \"398e7d66-1714-44d9-9a61-0883a4cc46c6\") "
Oct 29 22:40:42.772725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772649    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.772858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772820    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.772858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.772832    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.773065 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.773039    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:42.774173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.774151    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.774247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.774181    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398e7d66-1714-44d9-9a61-0883a4cc46c6-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.774247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.774198    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.774247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.774210    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.774247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.774222    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.775092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.775069    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.775185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.775087    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:42.775240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.775195    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:42.775294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.775286    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr" pod "398e7d66-1714-44d9-9a61-0883a4cc46c6" (UID: "398e7d66-1714-44d9-9a61-0883a4cc46c6"). InnerVolumeSpecName "kube-api-access-s89rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:42.875703 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.875672    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.875703 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.875698    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.875703 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.875708    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398e7d66-1714-44d9-9a61-0883a4cc46c6-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.876117 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:42.875716    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s89rr\" (UniqueName: \"kubernetes.io/projected/398e7d66-1714-44d9-9a61-0883a4cc46c6-kube-api-access-s89rr\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.444364 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.444333    4451 generic.go:356] "Generic (PLEG): container finished" podID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerID="2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775" exitCode=0
Oct 29 22:40:43.444570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.444390    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"
Oct 29 22:40:43.444570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.444431    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" event={"ID":"398e7d66-1714-44d9-9a61-0883a4cc46c6","Type":"ContainerDied","Data":"2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775"}
Oct 29 22:40:43.444570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.444493    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g" event={"ID":"398e7d66-1714-44d9-9a61-0883a4cc46c6","Type":"ContainerDied","Data":"f2e729160b715168bd5cca4620870396a85051bab17b74d26a1af02f6a05b965"}
Oct 29 22:40:43.444570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.444522    4451 scope.go:122] "RemoveContainer" containerID="2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775"
Oct 29 22:40:43.455217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.455202    4451 scope.go:122] "RemoveContainer" containerID="2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775"
Oct 29 22:40:43.455468 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:40:43.455450    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775\": container with ID starting with 2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775 not found: ID does not exist" containerID="2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775"
Oct 29 22:40:43.455520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.455476    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775"} err="failed to get container status \"2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775\": rpc error: code = NotFound desc = could not find container \"2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775\": container with ID starting with 2e1a3b9d68c2538bf50154a7b5770e8af330765b95c85078d0cc3c3729693775 not found: ID does not exist"
Oct 29 22:40:43.502958 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.502905    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"]
Oct 29 22:40:43.512227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:43.512206    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-zsn6g"]
Oct 29 22:40:44.164324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:44.164293    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" path="/var/lib/kubelet/pods/398e7d66-1714-44d9-9a61-0883a4cc46c6/volumes"
Oct 29 22:40:48.752181 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:40:48.752129    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:40:52.529651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:52.529613    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"]
Oct 29 22:40:52.530256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:52.529880    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy" containerID="cri-o://f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7" gracePeriod=30
Oct 29 22:40:57.840552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.840518    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:40:57.909080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909041    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909158    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909270    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909320    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909346    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909413    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909529    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909569    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909754    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo\") pod \"cc9540bf-3221-482d-8b21-8d6b535eb86c\" (UID: \"cc9540bf-3221-482d-8b21-8d6b535eb86c\") "
Oct 29 22:40:57.909843 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.909809    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:57.910067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910021    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:57.910213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910187    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:57.910412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910391    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:57.910568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910546    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:57.910843 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910822    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:57.910937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910851    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:57.910937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910867    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:57.910937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910898    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:57.910937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.910934    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9540bf-3221-482d-8b21-8d6b535eb86c-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:57.912789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.912754    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:57.914015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.913963    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "kube-api-access-qqp8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:57.914124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.914026    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:57.914363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:57.914343    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo" pod "cc9540bf-3221-482d-8b21-8d6b535eb86c" (UID: "cc9540bf-3221-482d-8b21-8d6b535eb86c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:58.019836 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.019799    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.019836 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.019834    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.019836 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.019851    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqp8q\" (UniqueName: \"kubernetes.io/projected/cc9540bf-3221-482d-8b21-8d6b535eb86c-kube-api-access-qqp8q\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.020180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.019866    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9540bf-3221-482d-8b21-8d6b535eb86c-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.622526 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.622485    4451 generic.go:356] "Generic (PLEG): container finished" podID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerID="f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7" exitCode=0
Oct 29 22:40:58.622698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.622549    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"
Oct 29 22:40:58.623409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.622571    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" event={"ID":"cc9540bf-3221-482d-8b21-8d6b535eb86c","Type":"ContainerDied","Data":"f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7"}
Oct 29 22:40:58.623520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.623449    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq" event={"ID":"cc9540bf-3221-482d-8b21-8d6b535eb86c","Type":"ContainerDied","Data":"0202b9e7c00b5fc566838a076b90d56465b6eb4a94fe0cfbab3f7305bca0a5f4"}
Oct 29 22:40:58.623520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.623484    4451 scope.go:122] "RemoveContainer" containerID="f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7"
Oct 29 22:40:58.637782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.637761    4451 scope.go:122] "RemoveContainer" containerID="f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7"
Oct 29 22:40:58.638199 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:40:58.638169    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7\": container with ID starting with f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7 not found: ID does not exist" containerID="f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7"
Oct 29 22:40:58.638280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.638210    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7"} err="failed to get container status \"f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7\": rpc error: code = NotFound desc = could not find container \"f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7\": container with ID starting with f25acae4f53947bfe3564d881987268a6e779f68bc93f804708d02a9a46268f7 not found: ID does not exist"
Oct 29 22:40:58.684678 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.684628    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"]
Oct 29 22:40:58.694484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:40:58.694455    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-7qslq"]
Oct 29 22:41:00.165426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:00.165394    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" path="/var/lib/kubelet/pods/cc9540bf-3221-482d-8b21-8d6b535eb86c/volumes"
Oct 29 22:41:18.756713 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:41:18.756674    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:41:25.758625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:25.758591    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"]
Oct 29 22:41:25.759346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:25.758871    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy" containerID="cri-o://ae58b824fd258918e431ace6e2bc1ecc9a8562b1900baa2bcdebf0a86c920cea" gracePeriod=30
Oct 29 22:41:25.769949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:25.769906    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"]
Oct 29 22:41:25.770374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:25.770346    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy" containerID="cri-o://4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f" gracePeriod=30
Oct 29 22:41:27.018790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:27.018740    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"]
Oct 29 22:41:27.019347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:27.019067    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy" containerID="cri-o://a5a38e0b5872326eb4ee0bf0cb8cd0a9f54da13f06bd3c6d9dc95f589fcf90c6" gracePeriod=30
Oct 29 22:41:31.072841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.072809    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:41:31.183517 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183480    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.183700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183577    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.183700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183620    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.183700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183643    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.183887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183757    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.183887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183820    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.184071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183952    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.184071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.183985    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.184181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184091    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.184181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184154    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data\") pod \"073a784f-7318-46b7-bead-566b6fa028ec\" (UID: \"073a784f-7318-46b7-bead-566b6fa028ec\") "
Oct 29 22:41:31.184302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184241    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.184302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184264    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.184658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184572    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.185073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184961    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a784f-7318-46b7-bead-566b6fa028ec-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.185073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.184987    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.185073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.185002    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.185073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.185017    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.185356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.185089    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.186455 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.186428    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.186608 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.186582    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "kube-api-access-p2bvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.186742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.186717    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.187872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.187837    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo" pod "073a784f-7318-46b7-bead-566b6fa028ec" (UID: "073a784f-7318-46b7-bead-566b6fa028ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.287390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.287356    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.287390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.287389    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.287569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.287398    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2bvp\" (UniqueName: \"kubernetes.io/projected/073a784f-7318-46b7-bead-566b6fa028ec-kube-api-access-p2bvp\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.287569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.287408    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a784f-7318-46b7-bead-566b6fa028ec-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.287569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.287416    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a784f-7318-46b7-bead-566b6fa028ec-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.294945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.294828    4451 generic.go:356] "Generic (PLEG): container finished" podID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerID="ae58b824fd258918e431ace6e2bc1ecc9a8562b1900baa2bcdebf0a86c920cea" exitCode=0
Oct 29 22:41:31.295100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.295030    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" event={"ID":"c36c9256-6d81-4b63-bb9d-9e9b5016f19a","Type":"ContainerDied","Data":"ae58b824fd258918e431ace6e2bc1ecc9a8562b1900baa2bcdebf0a86c920cea"}
Oct 29 22:41:31.296588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.296565    4451 generic.go:356] "Generic (PLEG): container finished" podID="073a784f-7318-46b7-bead-566b6fa028ec" containerID="4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f" exitCode=0
Oct 29 22:41:31.296711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.296634    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"
Oct 29 22:41:31.296711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.296652    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" event={"ID":"073a784f-7318-46b7-bead-566b6fa028ec","Type":"ContainerDied","Data":"4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f"}
Oct 29 22:41:31.296711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.296697    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr" event={"ID":"073a784f-7318-46b7-bead-566b6fa028ec","Type":"ContainerDied","Data":"ae7913f8dcd7cf08c3138978962f062cfa7fda641c08874202544dccbb6abe93"}
Oct 29 22:41:31.296834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.296723    4451 scope.go:122] "RemoveContainer" containerID="4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f"
Oct 29 22:41:31.310426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.310403    4451 scope.go:122] "RemoveContainer" containerID="4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f"
Oct 29 22:41:31.310786 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:41:31.310757    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f\": container with ID starting with 4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f not found: ID does not exist" containerID="4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f"
Oct 29 22:41:31.310899 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.310796    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f"} err="failed to get container status \"4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f\": rpc error: code = NotFound desc = could not find container \"4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f\": container with ID starting with 4572b22f1d1c9f870c38561f175919933b6c619872aa803b0d19348596d3527f not found: ID does not exist"
Oct 29 22:41:31.352227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.352191    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"]
Oct 29 22:41:31.358743 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.358220    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-ms7lr"]
Oct 29 22:41:31.453616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.453593    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:41:31.591232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591202    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591248    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591278    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591393    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591436    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591511    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591545    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591651    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591648    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.591841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591722    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.591841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591763    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket\") pod \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\" (UID: \"c36c9256-6d81-4b63-bb9d-9e9b5016f19a\") "
Oct 29 22:41:31.591995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591971    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.592119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.592098    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.592232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.591880    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.592491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.592472    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.592491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.592491    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.592617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.592501    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.592617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.592510    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.592617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.592518    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.593710 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.593688    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.593976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.593947    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.593976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.593954    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.594088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.593987    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd" pod "c36c9256-6d81-4b63-bb9d-9e9b5016f19a" (UID: "c36c9256-6d81-4b63-bb9d-9e9b5016f19a"). InnerVolumeSpecName "kube-api-access-vlvbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.701245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.695906    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.701245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.695955    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.701245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.695969    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.701245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:31.695983    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvbd\" (UniqueName: \"kubernetes.io/projected/c36c9256-6d81-4b63-bb9d-9e9b5016f19a-kube-api-access-vlvbd\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.203729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.203689    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073a784f-7318-46b7-bead-566b6fa028ec" path="/var/lib/kubelet/pods/073a784f-7318-46b7-bead-566b6fa028ec/volumes"
Oct 29 22:41:32.310131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.308960    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q" event={"ID":"c36c9256-6d81-4b63-bb9d-9e9b5016f19a","Type":"ContainerDied","Data":"a859575726aec5ae74eb9b5119ede0fa5869802eafde6fc29bf944746d664be6"}
Oct 29 22:41:32.310131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.309041    4451 scope.go:122] "RemoveContainer" containerID="ae58b824fd258918e431ace6e2bc1ecc9a8562b1900baa2bcdebf0a86c920cea"
Oct 29 22:41:32.310131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.309160    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"
Oct 29 22:41:32.324149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.322351    4451 generic.go:356] "Generic (PLEG): container finished" podID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerID="a5a38e0b5872326eb4ee0bf0cb8cd0a9f54da13f06bd3c6d9dc95f589fcf90c6" exitCode=0
Oct 29 22:41:32.324149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.322454    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" event={"ID":"8724b2c4-0f22-4d90-8b0e-e6d9527a06be","Type":"ContainerDied","Data":"a5a38e0b5872326eb4ee0bf0cb8cd0a9f54da13f06bd3c6d9dc95f589fcf90c6"}
Oct 29 22:41:32.367249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.367206    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"]
Oct 29 22:41:32.373019 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.372977    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-xgh4q"]
Oct 29 22:41:32.519435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.519405    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:41:32.631294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631259    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631300    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631438    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631605 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631552    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631624    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631719    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631763    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631793    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.631897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.631839    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket\") pod \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\" (UID: \"8724b2c4-0f22-4d90-8b0e-e6d9527a06be\") "
Oct 29 22:41:32.632606 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632032    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:32.632606 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632405    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:32.632788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632606    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:32.632788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632709    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:32.632788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632779    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.633017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632801    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.633017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.632814    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.633312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.633278    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:32.634085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.634055    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:32.634201 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.634178    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "kube-api-access-kvk2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:32.634316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.634297    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:32.634643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.634624    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token" pod "8724b2c4-0f22-4d90-8b0e-e6d9527a06be" (UID: "8724b2c4-0f22-4d90-8b0e-e6d9527a06be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:32.734515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.734478    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.734515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.734509    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.734515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.734518    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvk2c\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-kube-api-access-kvk2c\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.734515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.734527    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.734850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.734535    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.734850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:32.734542    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8724b2c4-0f22-4d90-8b0e-e6d9527a06be-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:33.014295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.014213    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"]
Oct 29 22:41:33.015206 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015176    4451 cpu_manager.go:414] "containerMap: removing container" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy"
Oct 29 22:41:33.015206 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015204    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015224    4451 cpu_manager.go:414] "containerMap: removing container" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015232    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015252    4451 cpu_manager.go:414] "containerMap: removing container" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015260    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015283    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015290    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015313    4451 cpu_manager.go:414] "containerMap: removing container" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy"
Oct 29 22:41:33.015418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015320    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy"
Oct 29 22:41:33.015823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015500    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9540bf-3221-482d-8b21-8d6b535eb86c" containerName="istio-proxy"
Oct 29 22:41:33.015823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015514    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" containerName="istio-proxy"
Oct 29 22:41:33.015823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015526    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="073a784f-7318-46b7-bead-566b6fa028ec" containerName="istio-proxy"
Oct 29 22:41:33.015823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015540    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" containerName="istio-proxy"
Oct 29 22:41:33.015823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.015552    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="398e7d66-1714-44d9-9a61-0883a4cc46c6" containerName="istio-proxy"
Oct 29 22:41:33.019942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.019905    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.057403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.057377    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"]
Oct 29 22:41:33.063955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.062576    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8d985a\"/\"openshift-service-ca.crt\""
Oct 29 22:41:33.063955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.062771    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8d985a\"/\"kube-root-ca.crt\""
Oct 29 22:41:33.063955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.062855    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8d985a\"/\"default-dockercfg-ldvnb\""
Oct 29 22:41:33.063955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.062873    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8d985a\"/\"test-model-8d985a-kserve-self-signed-certs\""
Oct 29 22:41:33.140627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.140591    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.140797 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.140656    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.140893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.140874    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.141002 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.140925    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.141002 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.140966    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.141103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.141036    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99jw\" (UniqueName: \"kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242179 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242143    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242198    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242238    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242264    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99jw\" (UniqueName: \"kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242369    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242439    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242594    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.242673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242621    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.243011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.242896    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.244728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.244698    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.245290 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.245268    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.276764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.276694    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99jw\" (UniqueName: \"kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw\") pod \"test-model-8d985a-kserve-5cb869c8cc-llw28\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") " pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.334297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.334258    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:33.334636 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.334614    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"
Oct 29 22:41:33.334636 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.334622    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds" event={"ID":"8724b2c4-0f22-4d90-8b0e-e6d9527a06be","Type":"ContainerDied","Data":"b0a0a0dba350798943b077117e5850b81406f7d8d932615c22ad4e8ef0173f5e"}
Oct 29 22:41:33.334784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.334686    4451 scope.go:122] "RemoveContainer" containerID="a5a38e0b5872326eb4ee0bf0cb8cd0a9f54da13f06bd3c6d9dc95f589fcf90c6"
Oct 29 22:41:33.447565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.447527    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"]
Oct 29 22:41:33.480824 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.471195    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-p9dds"]
Oct 29 22:41:33.827952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:33.826675    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"]
Oct 29 22:41:33.849465 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:41:33.849426    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bb87dcd_a9b6_464a_95da_a4e0f007e8a3.slice/crio-65dbd6cfa612fd639d75623be6cfa0d7d67148809248102490c6881bc4494f25 WatchSource:0}: Error finding container 65dbd6cfa612fd639d75623be6cfa0d7d67148809248102490c6881bc4494f25: Status 404 returned error can't find the container with id 65dbd6cfa612fd639d75623be6cfa0d7d67148809248102490c6881bc4494f25
Oct 29 22:41:34.164059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:34.163985    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8724b2c4-0f22-4d90-8b0e-e6d9527a06be" path="/var/lib/kubelet/pods/8724b2c4-0f22-4d90-8b0e-e6d9527a06be/volumes"
Oct 29 22:41:34.164475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:34.164460    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36c9256-6d81-4b63-bb9d-9e9b5016f19a" path="/var/lib/kubelet/pods/c36c9256-6d81-4b63-bb9d-9e9b5016f19a/volumes"
Oct 29 22:41:34.345645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:34.345612    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28" event={"ID":"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3","Type":"ContainerStarted","Data":"22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220"}
Oct 29 22:41:34.346045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:34.345653    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28" event={"ID":"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3","Type":"ContainerStarted","Data":"65dbd6cfa612fd639d75623be6cfa0d7d67148809248102490c6881bc4494f25"}
Oct 29 22:41:34.405873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:34.405821    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28" podStartSLOduration=2.4058070750000002 podStartE2EDuration="2.405807075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:34.401739005 +0000 UTC m=+1634.344697713" watchObservedRunningTime="2026-10-29 22:41:34.405807075 +0000 UTC m=+1634.348765779"
Oct 29 22:41:35.334803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:35.334764    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:43.345784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:43.343581    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:43.359673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:43.359647    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:43.413735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:43.413705    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:41:48.741560 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:41:48.741522    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:41:52.936556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:52.936518    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"]
Oct 29 22:41:52.936980 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:52.936712    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy" containerID="cri-o://6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5" gracePeriod=30
Oct 29 22:41:57.459712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.459678    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"]
Oct 29 22:41:57.463521 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.463496    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.487643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.487611    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a5af18-serving-cert\""
Oct 29 22:41:57.491578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.491544    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"]
Oct 29 22:41:57.553486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.553455    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.553666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.553503    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.553720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.553695    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lwp\" (UniqueName: \"kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.553827 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.553808    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.655454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.655422    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.655629 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.655522    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lwp\" (UniqueName: \"kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.655629 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.655570    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.655750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.655734    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.656385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.656361    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.658226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.658203    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.658503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.658480    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.675739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.675709    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lwp\" (UniqueName: \"kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp\") pod \"maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:57.778374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:57.778280    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:58.106537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.106509    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"]
Oct 29 22:41:58.106855 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:41:58.106833    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ccd5cb_61b6_4f5e_92da_787746ee51d5.slice/crio-e7701c45883b2ea5821f6c1f9c035d779d676ed74b718b7405c2d331e3c6ee99 WatchSource:0}: Error finding container e7701c45883b2ea5821f6c1f9c035d779d676ed74b718b7405c2d331e3c6ee99: Status 404 returned error can't find the container with id e7701c45883b2ea5821f6c1f9c035d779d676ed74b718b7405c2d331e3c6ee99
Oct 29 22:41:58.108804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.108785    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:41:58.199616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.199589    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:41:58.270141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.270107    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.270818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.270791    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.271509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.271492    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.272032 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.272013    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.272521 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.272501    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.274415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.271289    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:58.274547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.271960    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:58.274547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.272928    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:58.274675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.273071    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:58.274675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.274424    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.274780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.274691    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.274780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.274751    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.274880 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.274808    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token\") pod \"fb6af419-5ad8-4bdc-b316-51340757cd07\" (UID: \"fb6af419-5ad8-4bdc-b316-51340757cd07\") "
Oct 29 22:41:58.275389 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.275368    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.275474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.275392    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.275474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.275405    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6af419-5ad8-4bdc-b316-51340757cd07-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.275474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.275417    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.275804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.275785    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:58.276141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.276121    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:58.277046 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.277008    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "kube-api-access-xhwms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:58.280579 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.280557    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:58.280735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.280711    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token" pod "fb6af419-5ad8-4bdc-b316-51340757cd07" (UID: "fb6af419-5ad8-4bdc-b316-51340757cd07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:58.378865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.378271    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.378865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.378310    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwms\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-kube-api-access-xhwms\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.378865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.378327    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.378865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.378341    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6af419-5ad8-4bdc-b316-51340757cd07-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.378865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.378354    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6af419-5ad8-4bdc-b316-51340757cd07-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:41:58.531250 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.531216    4451 generic.go:356] "Generic (PLEG): container finished" podID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerID="6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5" exitCode=0
Oct 29 22:41:58.531650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.531257    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" event={"ID":"fb6af419-5ad8-4bdc-b316-51340757cd07","Type":"ContainerDied","Data":"6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5"}
Oct 29 22:41:58.531650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.531308    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"
Oct 29 22:41:58.531650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.531362    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657" event={"ID":"fb6af419-5ad8-4bdc-b316-51340757cd07","Type":"ContainerDied","Data":"c4aaa257b376978fb96aa70eaeb9e3b3918d28347c2902ffabe449ab94206973"}
Oct 29 22:41:58.531650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.531399    4451 scope.go:122] "RemoveContainer" containerID="6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5"
Oct 29 22:41:58.534663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.534642    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" event={"ID":"f4ccd5cb-61b6-4f5e-92da-787746ee51d5","Type":"ContainerStarted","Data":"e7701c45883b2ea5821f6c1f9c035d779d676ed74b718b7405c2d331e3c6ee99"}
Oct 29 22:41:58.544971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.544940    4451 scope.go:122] "RemoveContainer" containerID="6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5"
Oct 29 22:41:58.545295 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:41:58.545268    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5\": container with ID starting with 6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5 not found: ID does not exist" containerID="6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5"
Oct 29 22:41:58.545462 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.545304    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5"} err="failed to get container status \"6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5\": rpc error: code = NotFound desc = could not find container \"6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5\": container with ID starting with 6c0ca2d2e399364b58075cf7826fb409ae594374d59e172db87a8a2e11467ec5 not found: ID does not exist"
Oct 29 22:41:58.579260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.579233    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"]
Oct 29 22:41:58.582708 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:58.582671    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-bb657"]
Oct 29 22:41:59.543463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:59.543433    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" event={"ID":"f4ccd5cb-61b6-4f5e-92da-787746ee51d5","Type":"ContainerStarted","Data":"dead648187c56ed868a26300b9d60abf935cb820b7383ed7a324a56e58465da4"}
Oct 29 22:41:59.543855 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:59.543566    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:41:59.580434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:41:59.580388    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" podStartSLOduration=2.221339449 podStartE2EDuration="2.580374468s" totalImagesPullingTime="359.035019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:59.578349709 +0000 UTC m=+1659.521308423" watchObservedRunningTime="2026-10-29 22:41:59.580374468 +0000 UTC m=+1659.523333173"
Oct 29 22:42:00.165853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:00.165822    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" path="/var/lib/kubelet/pods/fb6af419-5ad8-4bdc-b316-51340757cd07/volumes"
Oct 29 22:42:02.665436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.665401    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"]
Oct 29 22:42:02.666112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.666096    4451 cpu_manager.go:414] "containerMap: removing container" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy"
Oct 29 22:42:02.666180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.666113    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy"
Oct 29 22:42:02.666246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.666235    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6af419-5ad8-4bdc-b316-51340757cd07" containerName="istio-proxy"
Oct 29 22:42:02.671175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.671155    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.675157 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.675130    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzkb\" (UniqueName: \"kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.675258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.675181    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.675376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.675296    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.675496 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.675477    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.694034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.694001    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c7d192-serving-cert\""
Oct 29 22:42:02.699952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.699921    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"]
Oct 29 22:42:02.777290 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.777245    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzkb\" (UniqueName: \"kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.777459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.777314    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.777459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.777370    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.777578 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:42:02.777482    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-serving-cert: secret "maas-api-e2e-inf-a-c7d192-serving-cert" not found
Oct 29 22:42:02.777578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.777516    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.777578 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:42:02.777574    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls podName:e30f6550-cc1b-49a3-8a87-a5d2d02317b8 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:03.277553044 +0000 UTC m=+1663.220511734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls") pod "maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" (UID: "e30f6550-cc1b-49a3-8a87-a5d2d02317b8") : secret "maas-api-e2e-inf-a-c7d192-serving-cert" not found
Oct 29 22:42:02.778340 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.778315    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.780257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.780234    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:02.797484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:02.797464    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzkb\" (UniqueName: \"kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:03.285866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:03.285825    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:03.288237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:03.288207    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:03.585842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:03.585805    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:03.931273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:03.927868    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"]
Oct 29 22:42:03.941260 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:42:03.941223    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30f6550_cc1b_49a3_8a87_a5d2d02317b8.slice/crio-88062ecb3a85c1f6b06d2ccb11d3a691576a7660af721adb19a002be5878ae00 WatchSource:0}: Error finding container 88062ecb3a85c1f6b06d2ccb11d3a691576a7660af721adb19a002be5878ae00: Status 404 returned error can't find the container with id 88062ecb3a85c1f6b06d2ccb11d3a691576a7660af721adb19a002be5878ae00
Oct 29 22:42:04.580437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:04.580397    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" event={"ID":"e30f6550-cc1b-49a3-8a87-a5d2d02317b8","Type":"ContainerStarted","Data":"d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501"}
Oct 29 22:42:04.580625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:04.580465    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" event={"ID":"e30f6550-cc1b-49a3-8a87-a5d2d02317b8","Type":"ContainerStarted","Data":"88062ecb3a85c1f6b06d2ccb11d3a691576a7660af721adb19a002be5878ae00"}
Oct 29 22:42:04.580625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:04.580494    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:04.619691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:04.619550    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" podStartSLOduration=2.334835799 podStartE2EDuration="2.619533121s" totalImagesPullingTime="284.697322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:04.61795473 +0000 UTC m=+1664.560913436" watchObservedRunningTime="2026-10-29 22:42:04.619533121 +0000 UTC m=+1664.562491826"
Oct 29 22:42:05.555353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:05.555327    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:42:10.594216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:10.594183    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:42:17.474314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.474272    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"]
Oct 29 22:42:17.479804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.479780    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.495930 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.495885    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"]
Oct 29 22:42:17.501553 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.501523    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5a69c9-serving-cert\""
Oct 29 22:42:17.595191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.595157    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.595362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.595294    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r9c\" (UniqueName: \"kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.595407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.595378    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.595454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.595438    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.697515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.697481    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.697718 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.697578    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r9c\" (UniqueName: \"kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.697718 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.697650    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.697718 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.697703    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.697947 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:42:17.697802    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-serving-cert: secret "maas-api-e2e-inf-a-5a69c9-serving-cert" not found
Oct 29 22:42:17.697947 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:42:17.697881    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls podName:e3e8c40a-295b-4149-8b4d-25086ce59925 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:18.197859823 +0000 UTC m=+1678.140818525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls") pod "maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" (UID: "e3e8c40a-295b-4149-8b4d-25086ce59925") : secret "maas-api-e2e-inf-a-5a69c9-serving-cert" not found
Oct 29 22:42:17.698277 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.698252    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.700416 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.700395    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:17.716640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:17.716601    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r9c\" (UniqueName: \"kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:18.206511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:18.206479    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:18.208924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:18.208887    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:18.398349 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:18.398311    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:18.743460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:18.743115    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"]
Oct 29 22:42:18.775956 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:42:18.774392    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:42:19.696333 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:19.696293    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" event={"ID":"e3e8c40a-295b-4149-8b4d-25086ce59925","Type":"ContainerStarted","Data":"4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b"}
Oct 29 22:42:19.696498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:19.696361    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:19.696498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:19.696383    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" event={"ID":"e3e8c40a-295b-4149-8b4d-25086ce59925","Type":"ContainerStarted","Data":"6c42170b6e771c8a9102e164c662d4110474195457c9dcf58d5d7a214afb75b7"}
Oct 29 22:42:19.732218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:19.732164    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" podStartSLOduration=2.240582436 podStartE2EDuration="2.732149858s" totalImagesPullingTime="491.567422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:19.730796399 +0000 UTC m=+1679.673755106" watchObservedRunningTime="2026-10-29 22:42:19.732149858 +0000 UTC m=+1679.675108562"
Oct 29 22:42:25.707679 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:25.707650    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:42:48.742966 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:42:48.742877    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:42:56.700541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.700504    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"]
Oct 29 22:42:56.705045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.705026    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.753328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.753291    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"]
Oct 29 22:42:56.814458 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.814426    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49mz\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.814712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.814532    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.814773 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.814743    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.814991 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.814935    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.815132 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.814997    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.815132 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.815094    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.815213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.815170    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.815247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.815209    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.815302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.815286    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.820226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.820199    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c7d192-openshift-default-dockercfg-v9rsq\""
Oct 29 22:42:56.916407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916377    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916490    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916518    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916545    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916573    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916602    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916648    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916707    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49mz\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.916818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.916773    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.917799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.917771    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.918071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.918051    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.918198 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.918141    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.918320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.918299    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.918369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.918338    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.919164 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.919135    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.919268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.919217    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.935642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.934523    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49mz\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:56.937327 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:56.937305    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:57.020189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:57.020121    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:57.338941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:57.338885    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"]
Oct 29 22:42:57.339369 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:42:57.339342    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c089e2_11bb_486d_a9e2_1eb04faa066f.slice/crio-656f5dc147b7c1409308bbfe38a2d778da8881ea4d6a6694bbf216af2f1d4f89 WatchSource:0}: Error finding container 656f5dc147b7c1409308bbfe38a2d778da8881ea4d6a6694bbf216af2f1d4f89: Status 404 returned error can't find the container with id 656f5dc147b7c1409308bbfe38a2d778da8881ea4d6a6694bbf216af2f1d4f89
Oct 29 22:42:57.342024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:57.341992    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:42:57.342102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:57.342057    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:42:57.342102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:57.342087    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:42:58.032515 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:58.032473    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" event={"ID":"49c089e2-11bb-486d-a9e2-1eb04faa066f","Type":"ContainerStarted","Data":"0c0390d9668f1b1ea482e5c08cd26876675d2638e652e7669ec6013c4a7e0a3a"}
Oct 29 22:42:58.033002 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:58.032536    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" event={"ID":"49c089e2-11bb-486d-a9e2-1eb04faa066f","Type":"ContainerStarted","Data":"656f5dc147b7c1409308bbfe38a2d778da8881ea4d6a6694bbf216af2f1d4f89"}
Oct 29 22:42:58.083389 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:58.083302    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podStartSLOduration=2.083284103 podStartE2EDuration="2.083284103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:58.081184815 +0000 UTC m=+1718.024143520" watchObservedRunningTime="2026-10-29 22:42:58.083284103 +0000 UTC m=+1718.026242806"
Oct 29 22:42:59.021241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:59.021203    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:42:59.023021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:59.022984    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:42:59.023169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:42:59.023064    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:43:00.021130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:00.021095    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.021543 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:00.021149    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:43:01.021280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:01.021241    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.021763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:01.021291    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:43:02.021937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:02.021268    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.021937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:02.021370    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:43:03.021554 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:03.021516    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:43:03.021859 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:03.021826    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:43:04.020856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:04.020820    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:43:04.028955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:04.028879    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:43:05.028596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:05.028561    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:43:05.030076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:05.030051    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:43:05.030749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:05.030729    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:43:08.184243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:08.184210    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"]
Oct 29 22:43:08.184692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:08.184397    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy" containerID="cri-o://d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a" gracePeriod=30
Oct 29 22:43:12.751849 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.751808    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"]
Oct 29 22:43:12.756642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.756616    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.784041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.784014    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8147a8-serving-cert\""
Oct 29 22:43:12.789006 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.788981    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"]
Oct 29 22:43:12.833267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.833235    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.833455 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.833432    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.833550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.833538    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk92k\" (UniqueName: \"kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.833595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.833561    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.944601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.940993    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk92k\" (UniqueName: \"kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.944601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.941055    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.944601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.941226    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.944601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.941411    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.946149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.946120    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.946436 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:12.946412    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-serving-cert: secret "maas-api-e2e-inf-b-8147a8-serving-cert" not found
Oct 29 22:43:12.946709 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:12.946691    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls podName:f80a04e9-d827-4c3e-b23c-ec35d86078d9 nodeName:}" failed. No retries permitted until 2026-10-29 22:43:13.446669619 +0000 UTC m=+1733.389628315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls") pod "maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" (UID: "f80a04e9-d827-4c3e-b23c-ec35d86078d9") : secret "maas-api-e2e-inf-b-8147a8-serving-cert" not found
Oct 29 22:43:12.947745 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.947723    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:12.974390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:12.974361    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk92k\" (UniqueName: \"kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:13.449226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.449167    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:13.451540 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.451518    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:13.460064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.460040    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:43:13.550081 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550053    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550106    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550146    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550249 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550206    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550406    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550512    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550546    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.550594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550548    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.551060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550685    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.551060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550727    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs\") pod \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\" (UID: \"7c9c5c4d-2fef-4281-bb15-8b8181ba9510\") "
Oct 29 22:43:13.551060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550714    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.551060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550863    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.551060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.550886    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:13.551372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.551084    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.551498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.551470    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.551625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.551502    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.551625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.551520    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.551625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.551536    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.551625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.551550    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.552662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.552637    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:13.553073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.553050    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:43:13.553073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.553057    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.553240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.553223    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc" pod "7c9c5c4d-2fef-4281-bb15-8b8181ba9510" (UID: "7c9c5c4d-2fef-4281-bb15-8b8181ba9510"). InnerVolumeSpecName "kube-api-access-mqsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:13.653356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.653018    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.653356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.653059    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.653356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.653072    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.653356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.653086    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqsvc\" (UniqueName: \"kubernetes.io/projected/7c9c5c4d-2fef-4281-bb15-8b8181ba9510-kube-api-access-mqsvc\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.695613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:13.695575    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:14.054171 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:43:14.054137    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80a04e9_d827_4c3e_b23c_ec35d86078d9.slice/crio-40e21d9f696ae65c783b932b370c1d0ba7a883d1a01287892f0c73c66b1d37ee WatchSource:0}: Error finding container 40e21d9f696ae65c783b932b370c1d0ba7a883d1a01287892f0c73c66b1d37ee: Status 404 returned error can't find the container with id 40e21d9f696ae65c783b932b370c1d0ba7a883d1a01287892f0c73c66b1d37ee
Oct 29 22:43:14.093091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.093056    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"]
Oct 29 22:43:14.164729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.164690    4451 generic.go:356] "Generic (PLEG): container finished" podID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerID="d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a" exitCode=0
Oct 29 22:43:14.164993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.164974    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"
Oct 29 22:43:14.177795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.176435    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" event={"ID":"f80a04e9-d827-4c3e-b23c-ec35d86078d9","Type":"ContainerStarted","Data":"40e21d9f696ae65c783b932b370c1d0ba7a883d1a01287892f0c73c66b1d37ee"}
Oct 29 22:43:14.177795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.176504    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" event={"ID":"7c9c5c4d-2fef-4281-bb15-8b8181ba9510","Type":"ContainerDied","Data":"d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a"}
Oct 29 22:43:14.177795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.176538    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r" event={"ID":"7c9c5c4d-2fef-4281-bb15-8b8181ba9510","Type":"ContainerDied","Data":"32808998ebaa0e36bc42085c01ad710f240a082ce8d84177836fd93173fbd2ba"}
Oct 29 22:43:14.177795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.176576    4451 scope.go:122] "RemoveContainer" containerID="d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a"
Oct 29 22:43:14.195870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.195827    4451 scope.go:122] "RemoveContainer" containerID="d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a"
Oct 29 22:43:14.196250 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:14.196218    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a\": container with ID starting with d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a not found: ID does not exist" containerID="d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a"
Oct 29 22:43:14.196380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.196261    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a"} err="failed to get container status \"d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a\": rpc error: code = NotFound desc = could not find container \"d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a\": container with ID starting with d81bf64d0cea0c3dee9743aabeda17d44ef677bb615c574c04285e8f7c02152a not found: ID does not exist"
Oct 29 22:43:14.258952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.258922    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"]
Oct 29 22:43:14.267550 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:14.267527    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-nr58r"]
Oct 29 22:43:15.173958 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:15.173906    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" event={"ID":"f80a04e9-d827-4c3e-b23c-ec35d86078d9","Type":"ContainerStarted","Data":"7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10"}
Oct 29 22:43:15.174398 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:15.174067    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:15.243330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:15.243281    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" podStartSLOduration=2.929636102 podStartE2EDuration="3.243266791s" totalImagesPullingTime="313.630689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:15.231781807 +0000 UTC m=+1735.174740527" watchObservedRunningTime="2026-10-29 22:43:15.243266791 +0000 UTC m=+1735.186225537"
Oct 29 22:43:16.165391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:16.165360    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" path="/var/lib/kubelet/pods/7c9c5c4d-2fef-4281-bb15-8b8181ba9510/volumes"
Oct 29 22:43:18.305900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.305868    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"]
Oct 29 22:43:18.306538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.306523    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy"
Oct 29 22:43:18.306625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.306540    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy"
Oct 29 22:43:18.306669 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.306664    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9c5c4d-2fef-4281-bb15-8b8181ba9510" containerName="istio-proxy"
Oct 29 22:43:18.310643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.310625    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.342906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.342884    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d3168-openshift-default-dockercfg-88hm9\""
Oct 29 22:43:18.359325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.359295    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"]
Oct 29 22:43:18.428967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.428927    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdpsc\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429104 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.428985    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429098    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429148    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429276 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429259    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429343    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429389 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429369    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429509    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.429591 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.429560    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.530833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.530805    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.530845    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.530962    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdpsc\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.530999    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531032    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531059    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531120    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531172    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531408 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531202    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531408 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531389    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531520 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531445    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531577    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531654    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.531765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.531678    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.533602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.533584    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.533933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.533876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.548868 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.548844    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdpsc\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.551508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.551480    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.624741 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.624712    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:18.750259 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:18.750222    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:43:18.949962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.949931    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"]
Oct 29 22:43:18.950745 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:43:18.950721    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode873e08f_6968_4cd5_84b0_350cd59ef5f8.slice/crio-fe9649889717f907a049ed5ec011bf4271d235e0768daba772e96f2ed6fd85b8 WatchSource:0}: Error finding container fe9649889717f907a049ed5ec011bf4271d235e0768daba772e96f2ed6fd85b8: Status 404 returned error can't find the container with id fe9649889717f907a049ed5ec011bf4271d235e0768daba772e96f2ed6fd85b8
Oct 29 22:43:18.953464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.953433    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:43:18.953528 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.953497    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:43:18.953528 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:18.953524    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:43:19.201820 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:19.201722    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" event={"ID":"e873e08f-6968-4cd5-84b0-350cd59ef5f8","Type":"ContainerStarted","Data":"c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb"}
Oct 29 22:43:19.201820 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:19.201800    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" event={"ID":"e873e08f-6968-4cd5-84b0-350cd59ef5f8","Type":"ContainerStarted","Data":"fe9649889717f907a049ed5ec011bf4271d235e0768daba772e96f2ed6fd85b8"}
Oct 29 22:43:19.258967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:19.258895    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podStartSLOduration=1.258881277 podStartE2EDuration="1.258881277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:19.245725963 +0000 UTC m=+1739.188684669" watchObservedRunningTime="2026-10-29 22:43:19.258881277 +0000 UTC m=+1739.201839981"
Oct 29 22:43:19.625368 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:19.625329    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:20.626558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:20.626517    4451 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:43:20.627215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:20.626573    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:43:21.185356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:21.185325    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:43:21.625149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:21.625109    4451 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:43:21.625353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:21.625184    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:43:22.395038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:22.390411    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"]
Oct 29 22:43:22.395038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:22.390634    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy" containerID="cri-o://c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911" gracePeriod=30
Oct 29 22:43:22.635812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:22.632333    4451 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:43:22.635812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:22.632401    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:43:23.625488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:23.625454    4451 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:43:23.625927 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:23.625521    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:43:24.117960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.117896    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"]
Oct 29 22:43:24.123737 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.123717    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.149767 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.149737    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e619b6-serving-cert\""
Oct 29 22:43:24.156419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.156385    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.156549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.156444    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.156549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.156475    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"]
Oct 29 22:43:24.156668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.156637    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2l8\" (UniqueName: \"kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.156726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.156668    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.258163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.258128    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2l8\" (UniqueName: \"kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.258163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.258163    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.258421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.258402    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.258486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.258453    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.258553 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:24.258522    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-serving-cert: secret "maas-api-e2e-inf-b-e619b6-serving-cert" not found
Oct 29 22:43:24.258620 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:24.258581    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls podName:ebab7eb5-9446-43fd-9210-9f33a470640a nodeName:}" failed. No retries permitted until 2026-10-29 22:43:24.758567474 +0000 UTC m=+1744.701526156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls") pod "maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" (UID: "ebab7eb5-9446-43fd-9210-9f33a470640a") : secret "maas-api-e2e-inf-b-e619b6-serving-cert" not found
Oct 29 22:43:24.258985 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.258962    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.260876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.260856    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.276814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.276793    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2l8\" (UniqueName: \"kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.625844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.625803    4451 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:43:24.626316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.625859    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:43:24.772183 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.772136    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:24.774687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:24.774661    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:25.037971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:25.037794    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:25.417158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:25.417125    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"]
Oct 29 22:43:25.418143 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:43:25.418116    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebab7eb5_9446_43fd_9210_9f33a470640a.slice/crio-d6676e502e81a0813c79ed45593d9beb72739cac5ebaef11feaa361c3b90164e WatchSource:0}: Error finding container d6676e502e81a0813c79ed45593d9beb72739cac5ebaef11feaa361c3b90164e: Status 404 returned error can't find the container with id d6676e502e81a0813c79ed45593d9beb72739cac5ebaef11feaa361c3b90164e
Oct 29 22:43:25.779337 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:25.779229    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:25.780987 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:25.780931    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:25.785548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:25.785519    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:43:26.264016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:26.263969    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" event={"ID":"ebab7eb5-9446-43fd-9210-9f33a470640a","Type":"ContainerStarted","Data":"970cd5ec8b1708398974e4a15a4b4e9fbf7cc3565c725df6703ab1fc4bfbf9e0"}
Oct 29 22:43:26.264186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:26.264033    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" event={"ID":"ebab7eb5-9446-43fd-9210-9f33a470640a","Type":"ContainerStarted","Data":"d6676e502e81a0813c79ed45593d9beb72739cac5ebaef11feaa361c3b90164e"}
Oct 29 22:43:26.264186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:26.264085    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:28.133476 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.133447    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:43:28.210428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.210325    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" podStartSLOduration=3.793775718 podStartE2EDuration="4.210309281s" totalImagesPullingTime="416.533563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:26.317752731 +0000 UTC m=+1746.260711436" watchObservedRunningTime="2026-10-29 22:43:28.210309281 +0000 UTC m=+1748.153267985"
Oct 29 22:43:28.224637 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.224603    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.224809 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.224759    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.224883 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.224814    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.224883 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.224845    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.225054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.224881    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.225054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.224988    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.225054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225035    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.225054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225039    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:28.225265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225077    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.225317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225263    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:28.225374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225309    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo\") pod \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\" (UID: \"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70\") "
Oct 29 22:43:28.225481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225461    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:28.225547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225477    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:28.225724 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.225699    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:28.226420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.226394    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.226530 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.226424    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.226530 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.226440    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.226530 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.226455    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.226530 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.226470    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.228627 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.228592    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:43:28.231069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.231043    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:28.231188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.231166    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "kube-api-access-7l6h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:28.231263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.231210    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token" pod "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" (UID: "ecfe7114-b8db-4fa6-8b35-6de9c1a95a70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:28.277535 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.277508    4451 generic.go:356] "Generic (PLEG): container finished" podID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerID="c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911" exitCode=0
Oct 29 22:43:28.277667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.277558    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"
Oct 29 22:43:28.277667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.277589    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" event={"ID":"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70","Type":"ContainerDied","Data":"c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911"}
Oct 29 22:43:28.277667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.277638    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6" event={"ID":"ecfe7114-b8db-4fa6-8b35-6de9c1a95a70","Type":"ContainerDied","Data":"d4015bd75c7c51ecc162fdeba68e522989988a8ccdd0c9aa383757f563216f07"}
Oct 29 22:43:28.277667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.277660    4451 scope.go:122] "RemoveContainer" containerID="c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911"
Oct 29 22:43:28.288739 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.288724    4451 scope.go:122] "RemoveContainer" containerID="c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911"
Oct 29 22:43:28.288984 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:28.288964    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911\": container with ID starting with c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911 not found: ID does not exist" containerID="c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911"
Oct 29 22:43:28.289047 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.288997    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911"} err="failed to get container status \"c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911\": rpc error: code = NotFound desc = could not find container \"c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911\": container with ID starting with c05047abbb198c0cd5ec8336a3a4cc1042d5afba6c5d51ce51b408b52851c911 not found: ID does not exist"
Oct 29 22:43:28.332165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.332132    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.332165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.332164    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.332382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.332178    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6h7\" (UniqueName: \"kubernetes.io/projected/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-kube-api-access-7l6h7\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.332382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.332193    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:28.335070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.335038    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"]
Oct 29 22:43:28.344568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:28.344547    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-lx7p6"]
Oct 29 22:43:30.165854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:30.165822    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" path="/var/lib/kubelet/pods/ecfe7114-b8db-4fa6-8b35-6de9c1a95a70/volumes"
Oct 29 22:43:32.279491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:32.279460    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:43:34.908572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.908489    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"]
Oct 29 22:43:34.909358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.909339    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy"
Oct 29 22:43:34.909358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.909358    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy"
Oct 29 22:43:34.909516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.909468    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfe7114-b8db-4fa6-8b35-6de9c1a95a70" containerName="istio-proxy"
Oct 29 22:43:34.914866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.914844    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:34.937792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.937765    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2a3ef0-serving-cert\""
Oct 29 22:43:34.943349 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:34.943323    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"]
Oct 29 22:43:35.046209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.046177    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhs5\" (UniqueName: \"kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.046376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.046228    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.046376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.046292    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.046450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.046416    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.147988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.147952    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhs5\" (UniqueName: \"kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.148713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.148672    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.148947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.148924    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.149242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.149211    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.149700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.149663    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.172652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.172507    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.178602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.178511    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.180394 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.180367    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhs5\" (UniqueName: \"kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5\") pod \"maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.230045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.230009    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:35.561964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:35.561930    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"]
Oct 29 22:43:35.562153 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:43:35.562129    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6207ed6_53f4_47cd_b253_907cdb512979.slice/crio-cf39a0dafdd0260d1ed4d57d0ecf9e52a887f016d025147b1285b5fcb53154a9 WatchSource:0}: Error finding container cf39a0dafdd0260d1ed4d57d0ecf9e52a887f016d025147b1285b5fcb53154a9: Status 404 returned error can't find the container with id cf39a0dafdd0260d1ed4d57d0ecf9e52a887f016d025147b1285b5fcb53154a9
Oct 29 22:43:36.380933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:36.376666    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" event={"ID":"f6207ed6-53f4-47cd-b253-907cdb512979","Type":"ContainerStarted","Data":"1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd"}
Oct 29 22:43:36.380933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:36.376733    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" event={"ID":"f6207ed6-53f4-47cd-b253-907cdb512979","Type":"ContainerStarted","Data":"cf39a0dafdd0260d1ed4d57d0ecf9e52a887f016d025147b1285b5fcb53154a9"}
Oct 29 22:43:36.380933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:36.377885    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:36.421263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:36.421214    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" podStartSLOduration=2.02859678 podStartE2EDuration="2.421198202s" totalImagesPullingTime="392.601422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:36.419721137 +0000 UTC m=+1756.362679843" watchObservedRunningTime="2026-10-29 22:43:36.421198202 +0000 UTC m=+1756.364156907"
Oct 29 22:43:38.572639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:38.572604    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"]
Oct 29 22:43:38.573152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:38.572855    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy" containerID="cri-o://e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68" gracePeriod=30
Oct 29 22:43:43.396088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.396044    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:43:43.853625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.853601    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:43:43.893118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893084    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893170    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893273 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893259    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893341    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.893440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893405    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893438    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893528    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893582    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893649 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893615    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893649 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893635    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.893758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893704    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy\") pod \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\" (UID: \"66fb1c3f-d992-460d-b1ff-ee45fa257c5a\") "
Oct 29 22:43:43.893758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.893718    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.894614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.894384    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.894614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.894524    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.894954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.894929    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.895066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.894959    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.895066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.894974    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.895066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.894988    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.896148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.896113    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:43.896247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.896226    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "kube-api-access-764st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:43.896389 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.896368    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:43:43.896451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.896420    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy" pod "66fb1c3f-d992-460d-b1ff-ee45fa257c5a" (UID: "66fb1c3f-d992-460d-b1ff-ee45fa257c5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.996533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.996496    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.996533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.996527    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.996533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.996539    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.996773 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.996551    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.996773 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:43.996563    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-764st\" (UniqueName: \"kubernetes.io/projected/66fb1c3f-d992-460d-b1ff-ee45fa257c5a-kube-api-access-764st\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:44.431038 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.431006    4451 generic.go:356] "Generic (PLEG): container finished" podID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerID="e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68" exitCode=0
Oct 29 22:43:44.431497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.431053    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"
Oct 29 22:43:44.431497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.431082    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" event={"ID":"66fb1c3f-d992-460d-b1ff-ee45fa257c5a","Type":"ContainerDied","Data":"e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68"}
Oct 29 22:43:44.431497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.431142    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz" event={"ID":"66fb1c3f-d992-460d-b1ff-ee45fa257c5a","Type":"ContainerDied","Data":"9ba68a507f0af4a7011253839b39e266eaa29139ac1e70d7937fe9f00da427c0"}
Oct 29 22:43:44.431497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.431173    4451 scope.go:122] "RemoveContainer" containerID="e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68"
Oct 29 22:43:44.441956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.441903    4451 scope.go:122] "RemoveContainer" containerID="e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68"
Oct 29 22:43:44.442222 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:44.442203    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68\": container with ID starting with e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68 not found: ID does not exist" containerID="e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68"
Oct 29 22:43:44.442289 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.442231    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68"} err="failed to get container status \"e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68\": rpc error: code = NotFound desc = could not find container \"e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68\": container with ID starting with e3404b9607d30f2d4378234edf7cdb9f396d3ec5a7e7721e09b30a982eea7f68 not found: ID does not exist"
Oct 29 22:43:44.480534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.480512    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"]
Oct 29 22:43:44.490472 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:44.490451    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-9vrlz"]
Oct 29 22:43:46.165268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:46.165238    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" path="/var/lib/kubelet/pods/66fb1c3f-d992-460d-b1ff-ee45fa257c5a/volumes"
Oct 29 22:43:48.741889 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:48.741854    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:43:52.702634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.702600    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"]
Oct 29 22:43:52.703553 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.703530    4451 cpu_manager.go:414] "containerMap: removing container" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy"
Oct 29 22:43:52.703664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.703554    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy"
Oct 29 22:43:52.703796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.703782    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fb1c3f-d992-460d-b1ff-ee45fa257c5a" containerName="istio-proxy"
Oct 29 22:43:52.709144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.709122    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.727723 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.727697    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-05417d-serving-cert\""
Oct 29 22:43:52.732388 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.732357    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"]
Oct 29 22:43:52.852127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.852087    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.852327 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.852297    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.852388 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.852354    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.852498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.852479    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckpqf\" (UniqueName: \"kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.954551 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.954452    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.955205 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.955172    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.955411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.955392    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.955607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.955590    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckpqf\" (UniqueName: \"kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.961683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.961655    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.965288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.965261    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.974867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.974833    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:52.981699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:52.981667    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckpqf\" (UniqueName: \"kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf\") pod \"maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:53.024437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:53.024408    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:53.335609 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:43:53.335576    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555d79bc_ed00_47ea_a390_0744438d6b82.slice/crio-eb6bdf442e1937f6605f1c43f0953df5fb823695acc72c91507ae980c7729751 WatchSource:0}: Error finding container eb6bdf442e1937f6605f1c43f0953df5fb823695acc72c91507ae980c7729751: Status 404 returned error can't find the container with id eb6bdf442e1937f6605f1c43f0953df5fb823695acc72c91507ae980c7729751
Oct 29 22:43:53.335853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:53.335813    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"]
Oct 29 22:43:53.503698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:53.503662    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g" event={"ID":"555d79bc-ed00-47ea-a390-0744438d6b82","Type":"ContainerStarted","Data":"eb6bdf442e1937f6605f1c43f0953df5fb823695acc72c91507ae980c7729751"}
Oct 29 22:43:53.633990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:53.629129    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"]
Oct 29 22:43:53.633990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:53.629426    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" containerID="cri-o://f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2" gracePeriod=30
Oct 29 22:43:54.501708 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.501675    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"]
Oct 29 22:43:54.506745 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.506718    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.513454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.513414    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g" event={"ID":"555d79bc-ed00-47ea-a390-0744438d6b82","Type":"ContainerStarted","Data":"ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec"}
Oct 29 22:43:54.513599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.513495    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:43:54.528202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.528181    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-6fc6bd-serving-cert\""
Oct 29 22:43:54.565282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.565247    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"]
Oct 29 22:43:54.591098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.591057    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.591267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.591189    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.591319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.591290    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.591665 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.591633    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vmv\" (UniqueName: \"kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.665619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.665569    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g" podStartSLOduration=2.293559698 podStartE2EDuration="2.665548298s" totalImagesPullingTime="371.9886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:54.66282591 +0000 UTC m=+1774.605784612" watchObservedRunningTime="2026-10-29 22:43:54.665548298 +0000 UTC m=+1774.608507002"
Oct 29 22:43:54.692664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.692636    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vmv\" (UniqueName: \"kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.692818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.692761    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.692885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.692852    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.692981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.692964    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.693802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.693781    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.695579 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.695553    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.695713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.695690    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.721719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.721697    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vmv\" (UniqueName: \"kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv\") pod \"maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:54.830119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:54.828931    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:55.173493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:55.173465    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"]
Oct 29 22:43:55.176387 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:43:55.176358    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1157cfc_7a7b_4686_a54b_a0942aea1d76.slice/crio-70e2def14598bc73c3cabf1e3629878b428de4826675e694fc2fa2ea157d8c91 WatchSource:0}: Error finding container 70e2def14598bc73c3cabf1e3629878b428de4826675e694fc2fa2ea157d8c91: Status 404 returned error can't find the container with id 70e2def14598bc73c3cabf1e3629878b428de4826675e694fc2fa2ea157d8c91
Oct 29 22:43:55.540008 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:55.537063    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg" event={"ID":"c1157cfc-7a7b-4686-a54b-a0942aea1d76","Type":"ContainerStarted","Data":"70e2def14598bc73c3cabf1e3629878b428de4826675e694fc2fa2ea157d8c91"}
Oct 29 22:43:56.545421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:56.545384    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg" event={"ID":"c1157cfc-7a7b-4686-a54b-a0942aea1d76","Type":"ContainerStarted","Data":"f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6"}
Oct 29 22:43:56.545879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:56.545602    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:43:56.618504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:56.618075    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg" podStartSLOduration=2.253982106 podStartE2EDuration="2.618055828s" totalImagesPullingTime="364.073722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:56.616465833 +0000 UTC m=+1776.559424558" watchObservedRunningTime="2026-10-29 22:43:56.618055828 +0000 UTC m=+1776.561014533"
Oct 29 22:43:58.059821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.059779    4451 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:43:58.060244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.059838    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:43:58.935949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.935898    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:43:58.979068 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979037    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979135    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979171    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979199    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979273    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979311    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979333    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979378    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979490    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data\") pod \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\" (UID: \"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0\") "
Oct 29 22:43:58.979658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979502    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:58.979658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979606    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:58.979828 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.979729    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:58.980025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980001    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:58.980182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980016    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:58.980447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980424    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:58.980523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980452    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:58.980523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980465    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:58.980523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980477    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:58.980523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.980492    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:58.982076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.982045    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:58.982157 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.982129    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:43:58.982201 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.982150    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:58.982236 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:58.982203    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p" pod "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" (UID: "6f0ef58b-50d5-42ec-a0e9-2f842d540cb0"). InnerVolumeSpecName "kube-api-access-5jd8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:59.082509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.082477    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:59.082509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.082511    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jd8p\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-kube-api-access-5jd8p\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:59.083004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.082526    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:59.083004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.082538    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:43:59.565386 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.565350    4451 generic.go:356] "Generic (PLEG): container finished" podID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerID="f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2" exitCode=0
Oct 29 22:43:59.565548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.565408    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" event={"ID":"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0","Type":"ContainerDied","Data":"f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2"}
Oct 29 22:43:59.565548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.565416    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"
Oct 29 22:43:59.565548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.565447    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7" event={"ID":"6f0ef58b-50d5-42ec-a0e9-2f842d540cb0","Type":"ContainerDied","Data":"a53b6a9e40780698d31f21fae0eafc2123ca8442f6de3b40d6f3cda5242d62df"}
Oct 29 22:43:59.565548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.565468    4451 scope.go:122] "RemoveContainer" containerID="f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2"
Oct 29 22:43:59.576453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.576435    4451 scope.go:122] "RemoveContainer" containerID="f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2"
Oct 29 22:43:59.576705 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:43:59.576686    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2\": container with ID starting with f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2 not found: ID does not exist" containerID="f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2"
Oct 29 22:43:59.576767 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.576717    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2"} err="failed to get container status \"f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2\": rpc error: code = NotFound desc = could not find container \"f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2\": container with ID starting with f0e815c1614a6067f26fb377eef171e5bfb65574fa12a5b72b747a4e167105a2 not found: ID does not exist"
Oct 29 22:43:59.622294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.622270    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"]
Oct 29 22:43:59.644852 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:43:59.644818    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-f7th7"]
Oct 29 22:44:00.167804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:00.167771    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" path="/var/lib/kubelet/pods/6f0ef58b-50d5-42ec-a0e9-2f842d540cb0/volumes"
Oct 29 22:44:00.543305 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:00.543222    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:44:02.557844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:02.557817    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 22:44:05.069753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.069719    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"]
Oct 29 22:44:05.070755 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.070735    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy"
Oct 29 22:44:05.070935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.070901    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy"
Oct 29 22:44:05.071237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.071223    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0ef58b-50d5-42ec-a0e9-2f842d540cb0" containerName="istio-proxy"
Oct 29 22:44:05.076978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.076956    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.111771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.111746    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fee294\"/\"kube-root-ca.crt\""
Oct 29 22:44:05.116096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.116067    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fee294\"/\"openshift-service-ca.crt\""
Oct 29 22:44:05.117596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.117558    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"]
Oct 29 22:44:05.121705 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.121677    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fee294\"/\"default-dockercfg-4n78j\""
Oct 29 22:44:05.121803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.121767    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fee294\"/\"test-model-fee294-kserve-self-signed-certs\""
Oct 29 22:44:05.177416 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.177376    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c2g4\" (UniqueName: \"kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.177592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.177553    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.177592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.177581    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.177760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.177607    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.177856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.177830    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.177940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.177874    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.279503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279439    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.279690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279511    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.279690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279539    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.279690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279661    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.279874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279690    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.279977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279936    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c2g4\" (UniqueName: \"kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.280063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.279976    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.280063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.280033    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.280275 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.280230    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.282142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.282123    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.282646 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.282627    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.315298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.315271    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c2g4\" (UniqueName: \"kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4\") pod \"test-model-fee294-kserve-5969ccbf7-sqw4d\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") " pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.395616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.395237    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:05.933106 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:44:05.933073    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb28aef5a_c6f0_415c_913c_35c8085afde6.slice/crio-4aaefe6d4b28dc53543a8afa94f7c77d04969ac6c2030a2818b6f8e800c8ac1f WatchSource:0}: Error finding container 4aaefe6d4b28dc53543a8afa94f7c77d04969ac6c2030a2818b6f8e800c8ac1f: Status 404 returned error can't find the container with id 4aaefe6d4b28dc53543a8afa94f7c77d04969ac6c2030a2818b6f8e800c8ac1f
Oct 29 22:44:05.933106 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:05.933082    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"]
Oct 29 22:44:06.624926 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:06.624888    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" event={"ID":"b28aef5a-c6f0-415c-913c-35c8085afde6","Type":"ContainerStarted","Data":"bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72"}
Oct 29 22:44:06.625310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:06.624960    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" event={"ID":"b28aef5a-c6f0-415c-913c-35c8085afde6","Type":"ContainerStarted","Data":"4aaefe6d4b28dc53543a8afa94f7c77d04969ac6c2030a2818b6f8e800c8ac1f"}
Oct 29 22:44:06.681994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:06.681947    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" podStartSLOduration=1.6819349749999999 podStartE2EDuration="1.681934975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:06.679388771 +0000 UTC m=+1786.622347476" watchObservedRunningTime="2026-10-29 22:44:06.681934975 +0000 UTC m=+1786.624893677"
Oct 29 22:44:07.397256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:07.397219    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:15.396662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:15.396628    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:15.414358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:15.414325    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:15.694238 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:44:15.694161    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:44:18.743010 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:44:18.742879    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:44:48.832937 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:44:48.824989    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:45:00.573666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.573628    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"]
Oct 29 22:45:00.578430 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.578404    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.596774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.596743    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"]
Oct 29 22:45:00.603117 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.603092    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.611867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.611841    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:45:00.619264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.619077    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"]
Oct 29 22:45:00.643676 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.643647    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"]
Oct 29 22:45:00.780905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.780619    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.780905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.780741    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ht29\" (UniqueName: \"kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.781541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.781290    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t872k\" (UniqueName: \"kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.781683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.781587    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.883592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.883507    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.883765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.883649    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ht29\" (UniqueName: \"kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.883877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.883861    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t872k\" (UniqueName: \"kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.883954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.883901    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.884188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.884167    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.884533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.884515    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.901975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.901947    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t872k\" (UniqueName: \"kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:00.902929 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.902895    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ht29\" (UniqueName: \"kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.906752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.906728    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:00.928336 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:00.928305    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:01.233168 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:01.233135    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"]
Oct 29 22:45:01.235546 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:45:01.235519    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761a7e6b_15d0_4930_8c8c_9b1472bb35c9.slice/crio-18c23ba3fb3eff1ba8629b932229b91c341685c8011a78cb550026938cd711ac WatchSource:0}: Error finding container 18c23ba3fb3eff1ba8629b932229b91c341685c8011a78cb550026938cd711ac: Status 404 returned error can't find the container with id 18c23ba3fb3eff1ba8629b932229b91c341685c8011a78cb550026938cd711ac
Oct 29 22:45:01.268372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:01.268345    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"]
Oct 29 22:45:01.269603 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:45:01.269576    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdbd0cd_d410_4267_bfdf_191ed25a7c12.slice/crio-b5cd785c9bec0dcd99a01b60d4e0aaa1c08f98cacc899f8d7706852740230ac6 WatchSource:0}: Error finding container b5cd785c9bec0dcd99a01b60d4e0aaa1c08f98cacc899f8d7706852740230ac6: Status 404 returned error can't find the container with id b5cd785c9bec0dcd99a01b60d4e0aaa1c08f98cacc899f8d7706852740230ac6
Oct 29 22:45:02.022200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:02.022168    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" event={"ID":"abdbd0cd-d410-4267-bfdf-191ed25a7c12","Type":"ContainerStarted","Data":"b5cd785c9bec0dcd99a01b60d4e0aaa1c08f98cacc899f8d7706852740230ac6"}
Oct 29 22:45:02.023503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:02.023465    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" event={"ID":"761a7e6b-15d0-4930-8c8c-9b1472bb35c9","Type":"ContainerStarted","Data":"18c23ba3fb3eff1ba8629b932229b91c341685c8011a78cb550026938cd711ac"}
Oct 29 22:45:05.046534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:05.046502    4451 generic.go:356] "Generic (PLEG): container finished" podID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerID="b938c90fb49783c1719f47aefe9dd8201d2269803095fd4bb6a6204c3642136a" exitCode=60
Oct 29 22:45:05.046869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:05.046580    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" event={"ID":"abdbd0cd-d410-4267-bfdf-191ed25a7c12","Type":"ContainerDied","Data":"b938c90fb49783c1719f47aefe9dd8201d2269803095fd4bb6a6204c3642136a"}
Oct 29 22:45:05.046869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:05.046827    4451 scope.go:122] "RemoveContainer" containerID="b938c90fb49783c1719f47aefe9dd8201d2269803095fd4bb6a6204c3642136a"
Oct 29 22:45:06.054886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:06.054850    4451 generic.go:356] "Generic (PLEG): container finished" podID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerID="0332279c8ceb7276aec36175753ae1479fd4e703277c65010cbc0d6f855d8a34" exitCode=60
Oct 29 22:45:06.055260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:06.054939    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" event={"ID":"abdbd0cd-d410-4267-bfdf-191ed25a7c12","Type":"ContainerDied","Data":"0332279c8ceb7276aec36175753ae1479fd4e703277c65010cbc0d6f855d8a34"}
Oct 29 22:45:06.055260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:06.054998    4451 scope.go:122] "RemoveContainer" containerID="b938c90fb49783c1719f47aefe9dd8201d2269803095fd4bb6a6204c3642136a"
Oct 29 22:45:06.055365 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:06.055280    4451 scope.go:122] "RemoveContainer" containerID="0332279c8ceb7276aec36175753ae1479fd4e703277c65010cbc0d6f855d8a34"
Oct 29 22:45:06.055664 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:06.055638    4451 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd_odh-ai-gateway-infra(abdbd0cd-d410-4267-bfdf-191ed25a7c12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12"
Oct 29 22:45:07.064126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.064098    4451 scope.go:122] "RemoveContainer" containerID="0332279c8ceb7276aec36175753ae1479fd4e703277c65010cbc0d6f855d8a34"
Oct 29 22:45:07.064621 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:07.064454    4451 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd_odh-ai-gateway-infra(abdbd0cd-d410-4267-bfdf-191ed25a7c12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12"
Oct 29 22:45:07.589900 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.589864    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"]
Oct 29 22:45:07.596014 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.595985    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.619573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.619541    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"]
Oct 29 22:45:07.624397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.624364    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2a3ef0\"/\"test-model-2a3ef0-kserve-self-signed-certs\""
Oct 29 22:45:07.624397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.624377    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2a3ef0\"/\"kube-root-ca.crt\""
Oct 29 22:45:07.624552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.624384    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2a3ef0\"/\"openshift-service-ca.crt\""
Oct 29 22:45:07.624673 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.624657    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2a3ef0\"/\"default-dockercfg-l68ww\""
Oct 29 22:45:07.680676 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.680641    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.680856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.680758    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89dl\" (UniqueName: \"kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.680856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.680801    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.681079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.681054    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.681166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.681094    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.681211 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.681197    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.784826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.784785    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.785121 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.785101    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89dl\" (UniqueName: \"kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.785272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.785257    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.785533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.785517    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.785647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.785634    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.785798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.785782    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.786509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.786485    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.786901 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.786881    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.790829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.790526    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.793663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.793117    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.793663 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.793302    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.808790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.808760    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89dl\" (UniqueName: \"kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl\") pod \"test-model-2a3ef0-kserve-7d4cc8cff7-q2292\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") " pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:07.922985 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:07.920210    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:08.338594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:08.338567    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"]
Oct 29 22:45:08.338958 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:45:08.338711    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c68168_f92c_4425_a8d5_843d540ebb90.slice/crio-ef49120e8bebc95eacf2590027f9a8a065fceb85e63585b206dd24a78fef2520 WatchSource:0}: Error finding container ef49120e8bebc95eacf2590027f9a8a065fceb85e63585b206dd24a78fef2520: Status 404 returned error can't find the container with id ef49120e8bebc95eacf2590027f9a8a065fceb85e63585b206dd24a78fef2520
Oct 29 22:45:09.083928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:09.083878    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292" event={"ID":"b2c68168-f92c-4425-a8d5-843d540ebb90","Type":"ContainerStarted","Data":"5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6"}
Oct 29 22:45:09.084077 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:09.083956    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292" event={"ID":"b2c68168-f92c-4425-a8d5-843d540ebb90","Type":"ContainerStarted","Data":"ef49120e8bebc95eacf2590027f9a8a065fceb85e63585b206dd24a78fef2520"}
Oct 29 22:45:09.123057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:09.122979    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292" podStartSLOduration=2.122960958 podStartE2EDuration="2.122960958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:09.120705648 +0000 UTC m=+1849.063664377" watchObservedRunningTime="2026-10-29 22:45:09.122960958 +0000 UTC m=+1849.065919663"
Oct 29 22:45:09.924934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:09.922007    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:12.104021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:12.103985    4451 generic.go:356] "Generic (PLEG): container finished" podID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerID="74ee2ac4319ad8038d4f137920a1927c25fb96569bcc33955fb3eabd3d3c6645" exitCode=60
Oct 29 22:45:12.104446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:12.104090    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" event={"ID":"761a7e6b-15d0-4930-8c8c-9b1472bb35c9","Type":"ContainerDied","Data":"74ee2ac4319ad8038d4f137920a1927c25fb96569bcc33955fb3eabd3d3c6645"}
Oct 29 22:45:12.104446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:12.104374    4451 scope.go:122] "RemoveContainer" containerID="74ee2ac4319ad8038d4f137920a1927c25fb96569bcc33955fb3eabd3d3c6645"
Oct 29 22:45:13.113024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:13.112986    4451 generic.go:356] "Generic (PLEG): container finished" podID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerID="8ab4f21583b8b06df5bf2e19379e7f03fd581d9851c84e4e799e4d4096301dad" exitCode=60
Oct 29 22:45:13.113421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:13.113070    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" event={"ID":"761a7e6b-15d0-4930-8c8c-9b1472bb35c9","Type":"ContainerDied","Data":"8ab4f21583b8b06df5bf2e19379e7f03fd581d9851c84e4e799e4d4096301dad"}
Oct 29 22:45:13.113421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:13.113131    4451 scope.go:122] "RemoveContainer" containerID="74ee2ac4319ad8038d4f137920a1927c25fb96569bcc33955fb3eabd3d3c6645"
Oct 29 22:45:13.113524 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:13.113430    4451 scope.go:122] "RemoveContainer" containerID="8ab4f21583b8b06df5bf2e19379e7f03fd581d9851c84e4e799e4d4096301dad"
Oct 29 22:45:13.113775 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:13.113753    4451 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59_odh-ai-gateway-infra(761a7e6b-15d0-4930-8c8c-9b1472bb35c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9"
Oct 29 22:45:14.121216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:14.121187    4451 scope.go:122] "RemoveContainer" containerID="8ab4f21583b8b06df5bf2e19379e7f03fd581d9851c84e4e799e4d4096301dad"
Oct 29 22:45:14.121585 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:14.121455    4451 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59_odh-ai-gateway-infra(761a7e6b-15d0-4930-8c8c-9b1472bb35c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9"
Oct 29 22:45:17.156990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:17.156962    4451 scope.go:122] "RemoveContainer" containerID="0332279c8ceb7276aec36175753ae1479fd4e703277c65010cbc0d6f855d8a34"
Oct 29 22:45:17.920979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:17.920945    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:17.933898 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:17.933870    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:18.155580 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:18.155551    4451 generic.go:356] "Generic (PLEG): container finished" podID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerID="f21888e1b340209294e13655427a2f120322329cf9ae7bf05639b3d57019420f" exitCode=60
Oct 29 22:45:18.166946 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:18.166899    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" event={"ID":"abdbd0cd-d410-4267-bfdf-191ed25a7c12","Type":"ContainerDied","Data":"f21888e1b340209294e13655427a2f120322329cf9ae7bf05639b3d57019420f"}
Oct 29 22:45:18.167407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:18.166977    4451 scope.go:122] "RemoveContainer" containerID="0332279c8ceb7276aec36175753ae1479fd4e703277c65010cbc0d6f855d8a34"
Oct 29 22:45:18.167480 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:18.167407    4451 scope.go:122] "RemoveContainer" containerID="f21888e1b340209294e13655427a2f120322329cf9ae7bf05639b3d57019420f"
Oct 29 22:45:18.167788 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:18.167760    4451 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd_odh-ai-gateway-infra(abdbd0cd-d410-4267-bfdf-191ed25a7c12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12"
Oct 29 22:45:18.173091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:18.173038    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:45:18.807934 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:18.806181    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:45:19.145090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.145052    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"]
Oct 29 22:45:19.145675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.145644    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy" containerID="cri-o://c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb" gracePeriod=30
Oct 29 22:45:19.225523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.225480    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"]
Oct 29 22:45:19.705376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.705343    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:19.722074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.721992    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k\" (UniqueName: \"kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k\") pod \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") "
Oct 29 22:45:19.722233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.722154    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca\") pod \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\" (UID: \"abdbd0cd-d410-4267-bfdf-191ed25a7c12\") "
Oct 29 22:45:19.722596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.722571    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca" pod "abdbd0cd-d410-4267-bfdf-191ed25a7c12" (UID: "abdbd0cd-d410-4267-bfdf-191ed25a7c12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:19.722775 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.722754    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdbd0cd-d410-4267-bfdf-191ed25a7c12-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:19.725017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.724990    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k" pod "abdbd0cd-d410-4267-bfdf-191ed25a7c12" (UID: "abdbd0cd-d410-4267-bfdf-191ed25a7c12"). InnerVolumeSpecName "kube-api-access-t872k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:19.825086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:19.825050    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t872k\" (UniqueName: \"kubernetes.io/projected/abdbd0cd-d410-4267-bfdf-191ed25a7c12-kube-api-access-t872k\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:20.208981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:20.208947    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd" event={"ID":"abdbd0cd-d410-4267-bfdf-191ed25a7c12","Type":"ContainerDied","Data":"b5cd785c9bec0dcd99a01b60d4e0aaa1c08f98cacc899f8d7706852740230ac6"}
Oct 29 22:45:20.208981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:20.208976    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"
Oct 29 22:45:20.209220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:20.209012    4451 scope.go:122] "RemoveContainer" containerID="f21888e1b340209294e13655427a2f120322329cf9ae7bf05639b3d57019420f"
Oct 29 22:45:20.261180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:20.261153    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"]
Oct 29 22:45:20.264782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:20.264755    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888565-ttmtd"]
Oct 29 22:45:22.166568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:22.166533    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" path="/var/lib/kubelet/pods/abdbd0cd-d410-4267-bfdf-191ed25a7c12/volumes"
Oct 29 22:45:23.472628 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:23.472596    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"]
Oct 29 22:45:23.473096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:23.472789    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy" containerID="cri-o://3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8" gracePeriod=30
Oct 29 22:45:24.525052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.525020    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:45:24.623460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623412    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.623639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623497    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.623639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623564    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.623810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623705    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.623810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623793    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.623962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623851    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.623962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623952    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.624080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.623996    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.624134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.624100    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs\") pod \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\" (UID: \"e873e08f-6968-4cd5-84b0-350cd59ef5f8\") "
Oct 29 22:45:24.648230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.646771    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:24.648230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.647106    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:24.648509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.648310    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:24.648509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.648396    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:24.648633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.648521    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:24.663317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.663260    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:24.666178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.666107    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:24.666178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.666118    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "kube-api-access-bdpsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:24.668435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.668386    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo" pod "e873e08f-6968-4cd5-84b0-350cd59ef5f8" (UID: "e873e08f-6968-4cd5-84b0-350cd59ef5f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:45:24.727753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727715    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727757    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727772    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727785    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727797    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727810    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727824    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727837    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e873e08f-6968-4cd5-84b0-350cd59ef5f8-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.727989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:24.727851    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdpsc\" (UniqueName: \"kubernetes.io/projected/e873e08f-6968-4cd5-84b0-350cd59ef5f8-kube-api-access-bdpsc\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.156966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.156898    4451 scope.go:122] "RemoveContainer" containerID="8ab4f21583b8b06df5bf2e19379e7f03fd581d9851c84e4e799e4d4096301dad"
Oct 29 22:45:25.277154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.277122    4451 generic.go:356] "Generic (PLEG): container finished" podID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerID="c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb" exitCode=0
Oct 29 22:45:25.277301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.277176    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"
Oct 29 22:45:25.277301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.277199    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" event={"ID":"e873e08f-6968-4cd5-84b0-350cd59ef5f8","Type":"ContainerDied","Data":"c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb"}
Oct 29 22:45:25.277301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.277260    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz" event={"ID":"e873e08f-6968-4cd5-84b0-350cd59ef5f8","Type":"ContainerDied","Data":"fe9649889717f907a049ed5ec011bf4271d235e0768daba772e96f2ed6fd85b8"}
Oct 29 22:45:25.277301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.277292    4451 scope.go:122] "RemoveContainer" containerID="c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb"
Oct 29 22:45:25.288768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.288747    4451 scope.go:122] "RemoveContainer" containerID="c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb"
Oct 29 22:45:25.289095 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:25.289073    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb\": container with ID starting with c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb not found: ID does not exist" containerID="c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb"
Oct 29 22:45:25.289182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.289111    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb"} err="failed to get container status \"c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb\": rpc error: code = NotFound desc = could not find container \"c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb\": container with ID starting with c9cbab24bd552fa0a4399adfe3d7e9d3393ed10e43e8c983cb2d81782182e5cb not found: ID does not exist"
Oct 29 22:45:25.333787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.333755    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"]
Oct 29 22:45:25.360531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:25.346115    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4g7sz"]
Oct 29 22:45:26.166735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:26.166696    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" path="/var/lib/kubelet/pods/e873e08f-6968-4cd5-84b0-350cd59ef5f8/volumes"
Oct 29 22:45:26.286212 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:26.286177    4451 generic.go:356] "Generic (PLEG): container finished" podID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerID="ac2db41c7006a8c04be472d1822774496472687649437c912edded35b8ccd132" exitCode=60
Oct 29 22:45:26.286396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:26.286269    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" event={"ID":"761a7e6b-15d0-4930-8c8c-9b1472bb35c9","Type":"ContainerDied","Data":"ac2db41c7006a8c04be472d1822774496472687649437c912edded35b8ccd132"}
Oct 29 22:45:26.286396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:26.286346    4451 scope.go:122] "RemoveContainer" containerID="8ab4f21583b8b06df5bf2e19379e7f03fd581d9851c84e4e799e4d4096301dad"
Oct 29 22:45:26.286852 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:26.286827    4451 scope.go:122] "RemoveContainer" containerID="ac2db41c7006a8c04be472d1822774496472687649437c912edded35b8ccd132"
Oct 29 22:45:26.287310 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:26.287281    4451 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59_odh-ai-gateway-infra(761a7e6b-15d0-4930-8c8c-9b1472bb35c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9"
Oct 29 22:45:27.329194 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.329157    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"]
Oct 29 22:45:27.513737 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.513707    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:27.674760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.674681    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca\") pod \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") "
Oct 29 22:45:27.674892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.674812    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29\" (UniqueName: \"kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29\") pod \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\" (UID: \"761a7e6b-15d0-4930-8c8c-9b1472bb35c9\") "
Oct 29 22:45:27.675155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.675130    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca" pod "761a7e6b-15d0-4930-8c8c-9b1472bb35c9" (UID: "761a7e6b-15d0-4930-8c8c-9b1472bb35c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:27.675538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.675521    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-maas-api-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.676947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.676904    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29" pod "761a7e6b-15d0-4930-8c8c-9b1472bb35c9" (UID: "761a7e6b-15d0-4930-8c8c-9b1472bb35c9"). InnerVolumeSpecName "kube-api-access-5ht29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:27.777403 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:27.777372    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ht29\" (UniqueName: \"kubernetes.io/projected/761a7e6b-15d0-4930-8c8c-9b1472bb35c9-kube-api-access-5ht29\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.304863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.304830    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"
Oct 29 22:45:28.305089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.304862    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59" event={"ID":"761a7e6b-15d0-4930-8c8c-9b1472bb35c9","Type":"ContainerDied","Data":"18c23ba3fb3eff1ba8629b932229b91c341685c8011a78cb550026938cd711ac"}
Oct 29 22:45:28.305089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.304941    4451 scope.go:122] "RemoveContainer" containerID="ac2db41c7006a8c04be472d1822774496472687649437c912edded35b8ccd132"
Oct 29 22:45:28.352393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.352359    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"]
Oct 29 22:45:28.356410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.356387    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888565-n9k59"]
Oct 29 22:45:28.783604 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.783576    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:45:28.906488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906450    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906562    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906598    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906685    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906783    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906814    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906846    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906873    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.906940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.906905    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.907210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.907042    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs\") pod \"72ab741f-a370-4992-b61e-b3b1866fdeeb\" (UID: \"72ab741f-a370-4992-b61e-b3b1866fdeeb\") "
Oct 29 22:45:28.907210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.907115    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.907210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.907127    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.908721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.907706    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.908721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.908127    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.908721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.908149    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.908721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.908163    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.908721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.908176    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.908721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.908684    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:28.909464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.909439    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.909617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.909586    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "kube-api-access-ffsgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:28.909823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.909803    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:45:28.910237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:28.910214    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token" pod "72ab741f-a370-4992-b61e-b3b1866fdeeb" (UID: "72ab741f-a370-4992-b61e-b3b1866fdeeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:29.010148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.010063    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.010148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.010092    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ab741f-a370-4992-b61e-b3b1866fdeeb-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.010148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.010101    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.010148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.010110    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ab741f-a370-4992-b61e-b3b1866fdeeb-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.010148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.010119    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsgl\" (UniqueName: \"kubernetes.io/projected/72ab741f-a370-4992-b61e-b3b1866fdeeb-kube-api-access-ffsgl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.330932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.330513    4451 generic.go:356] "Generic (PLEG): container finished" podID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerID="3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8" exitCode=0
Oct 29 22:45:29.330932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.330614    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" event={"ID":"72ab741f-a370-4992-b61e-b3b1866fdeeb","Type":"ContainerDied","Data":"3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8"}
Oct 29 22:45:29.330932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.330666    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx" event={"ID":"72ab741f-a370-4992-b61e-b3b1866fdeeb","Type":"ContainerDied","Data":"a1a240bb273ed2f18b8a334f49822b7d2f5bf5e1dec2210d43269f62be6c4851"}
Oct 29 22:45:29.330932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.330695    4451 scope.go:122] "RemoveContainer" containerID="3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8"
Oct 29 22:45:29.330932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.330835    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"
Oct 29 22:45:29.374009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.373978    4451 scope.go:122] "RemoveContainer" containerID="3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8"
Oct 29 22:45:29.374696 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:29.374661    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8\": container with ID starting with 3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8 not found: ID does not exist" containerID="3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8"
Oct 29 22:45:29.374798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.374700    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8"} err="failed to get container status \"3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8\": rpc error: code = NotFound desc = could not find container \"3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8\": container with ID starting with 3938900e6527347863f354de5ba945d14ae8f7a70ec4c4932124ccb94676d3d8 not found: ID does not exist"
Oct 29 22:45:29.429933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.428317    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"]
Oct 29 22:45:29.439971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:29.438297    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-g26hx"]
Oct 29 22:45:30.170871 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:30.170831    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" path="/var/lib/kubelet/pods/72ab741f-a370-4992-b61e-b3b1866fdeeb/volumes"
Oct 29 22:45:30.171573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:30.171546    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" path="/var/lib/kubelet/pods/761a7e6b-15d0-4930-8c8c-9b1472bb35c9/volumes"
Oct 29 22:45:32.134497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.134459    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"]
Oct 29 22:45:32.135642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135612    4451 cpu_manager.go:414] "containerMap: removing container" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.135642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135642    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135659    4451 cpu_manager.go:414] "containerMap: removing container" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135666    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135683    4451 cpu_manager.go:414] "containerMap: removing container" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135692    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135711    4451 cpu_manager.go:414] "containerMap: removing container" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135719    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135749    4451 cpu_manager.go:414] "containerMap: removing container" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135756    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135801    4451 cpu_manager.go:414] "containerMap: removing container" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:32.135848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.135808    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136004    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136022    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136042    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136056    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136067    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ab741f-a370-4992-b61e-b3b1866fdeeb" containerName="istio-proxy"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136076    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:32.136413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.136085    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="e873e08f-6968-4cd5-84b0-350cd59ef5f8" containerName="istio-proxy"
Oct 29 22:45:32.145475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.145451    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.168233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.167001    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"auth-test-model-9aafe6-kserve-self-signed-certs\""
Oct 29 22:45:32.168233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.167108    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"openshift-service-ca.crt\""
Oct 29 22:45:32.168233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.167191    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"kube-root-ca.crt\""
Oct 29 22:45:32.175864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.175832    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"default-dockercfg-75x84\""
Oct 29 22:45:32.180721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.180694    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"]
Oct 29 22:45:32.273279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.273248    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.273471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.273306    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.273471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.273447    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.273570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.273534    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.273614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.273566    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwdh\" (UniqueName: \"kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.273884 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.273866    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.383216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.383177    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.383526 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.383487    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.384163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.384140    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.384355 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.384336    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.384528 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.384485    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.384640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.384625    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwdh\" (UniqueName: \"kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.384950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.383858    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.389061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.385323    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.390363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.390338    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.392776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.392753    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.415436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.415399    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.426805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.425262    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwdh\" (UniqueName: \"kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh\") pod \"auth-test-model-9aafe6-kserve-56db74b869-bppzh\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.486685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.486653    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:32.834224 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:45:32.834188    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3ac0379_0e59_4ea2_a8b0_3a1cd844441b.slice/crio-3c86966e1d71f941992d3b885fec56d09053c2cf0b530d8244cd1f6af556e320 WatchSource:0}: Error finding container 3c86966e1d71f941992d3b885fec56d09053c2cf0b530d8244cd1f6af556e320: Status 404 returned error can't find the container with id 3c86966e1d71f941992d3b885fec56d09053c2cf0b530d8244cd1f6af556e320
Oct 29 22:45:32.835031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:32.834779    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"]
Oct 29 22:45:33.360623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:33.360589    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh" event={"ID":"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b","Type":"ContainerStarted","Data":"2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3"}
Oct 29 22:45:33.360623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:33.360629    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh" event={"ID":"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b","Type":"ContainerStarted","Data":"3c86966e1d71f941992d3b885fec56d09053c2cf0b530d8244cd1f6af556e320"}
Oct 29 22:45:33.405092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:33.405033    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh" podStartSLOduration=1.405015588 podStartE2EDuration="1.405015588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:33.401860231 +0000 UTC m=+1873.344818938" watchObservedRunningTime="2026-10-29 22:45:33.405015588 +0000 UTC m=+1873.347974292"
Oct 29 22:45:33.487885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:33.487854    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:42.340794 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.340760    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"]
Oct 29 22:45:42.341474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.341457    4451 cpu_manager.go:414] "containerMap: removing container" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:42.341518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.341476    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:42.341518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.341501    4451 cpu_manager.go:414] "containerMap: removing container" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:42.341518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.341506    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761a7e6b-15d0-4930-8c8c-9b1472bb35c9" containerName="cleanup"
Oct 29 22:45:42.341674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.341618    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdbd0cd-d410-4267-bfdf-191ed25a7c12" containerName="cleanup"
Oct 29 22:45:42.348082 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.348062    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.366114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.366088    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"]
Oct 29 22:45:42.474491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.474455    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.474666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.474515    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.474666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.474583    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.474753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.474697    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.474753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.474731    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.474846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.474831    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2gc\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.475031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.475017    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.475078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.475059    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.475180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.475165    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.488030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.488000    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:42.501410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.501378    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:42.576086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576051    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576089    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576112    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576135    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576202    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2gc\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576383    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576479 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576436    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576482    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576554    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576707 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576590    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576707 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576652    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.576838 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576816    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.577010 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.576990    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.577374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.577351    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.578497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.578475    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.578754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.578736    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.594960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.594876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.596351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.596325    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2gc\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.665504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.665473    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:42.991405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.991375    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"]
Oct 29 22:45:42.992449 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:45:42.992424    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adcf1d7_002a_4c06_9f78_0b5af2347bcf.slice/crio-12850dc823ce7a024cfda907a3d26ee5ad3098e230170829e182f70fe7b276e3 WatchSource:0}: Error finding container 12850dc823ce7a024cfda907a3d26ee5ad3098e230170829e182f70fe7b276e3: Status 404 returned error can't find the container with id 12850dc823ce7a024cfda907a3d26ee5ad3098e230170829e182f70fe7b276e3
Oct 29 22:45:42.995340 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.995306    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:45:42.995431 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.995373    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:45:42.995431 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:42.995401    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:45:43.433547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:43.433512    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" event={"ID":"6adcf1d7-002a-4c06-9f78-0b5af2347bcf","Type":"ContainerStarted","Data":"d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27"}
Oct 29 22:45:43.434055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:43.433573    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" event={"ID":"6adcf1d7-002a-4c06-9f78-0b5af2347bcf","Type":"ContainerStarted","Data":"12850dc823ce7a024cfda907a3d26ee5ad3098e230170829e182f70fe7b276e3"}
Oct 29 22:45:43.446652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:43.446622    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:45:43.482463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:43.482415    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podStartSLOduration=1.482401526 podStartE2EDuration="1.482401526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:43.471140379 +0000 UTC m=+1883.414099084" watchObservedRunningTime="2026-10-29 22:45:43.482401526 +0000 UTC m=+1883.425360230"
Oct 29 22:45:43.665838 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:43.665795    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:44.667131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:44.667091    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:45:44.667522 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:44.667146    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:45:45.666536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:45.666489    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:45:45.666711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:45.666557    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:45:46.666837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:46.666800    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:45:46.667237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:46.666857    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:45:47.665863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:47.665820    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:45:47.666122 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:47.665904    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:45:48.666197 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:48.666157    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:45:48.666691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:48.666210    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:45:48.750953 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:45:48.750310    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:45:49.680468 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:49.680438    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:49.680872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:49.680825    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:45:49.681950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:45:49.681924    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:46:18.746056 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:46:18.746010    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:46:27.961164 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:27.961113    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"]
Oct 29 22:46:38.687509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:38.687431    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"]
Oct 29 22:46:38.687970 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:38.687628    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy" containerID="cri-o://2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148" gracePeriod=30
Oct 29 22:46:42.988091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:42.988038    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28" podUID="7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" containerName="main" containerID="cri-o://22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220" gracePeriod=45
Oct 29 22:46:43.271779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.271752    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:46:43.318555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318524    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home\") pod \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") "
Oct 29 22:46:43.318734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318561    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir\") pod \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") "
Oct 29 22:46:43.318734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318621    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw\" (UniqueName: \"kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw\") pod \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") "
Oct 29 22:46:43.318865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318805    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home" pod "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" (UID: "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:43.318865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318841    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm\") pod \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") "
Oct 29 22:46:43.319000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318894    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir" pod "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" (UID: "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:43.319000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318902    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache\") pod \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") "
Oct 29 22:46:43.319000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.318978    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs\") pod \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\" (UID: \"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3\") "
Oct 29 22:46:43.319242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.319219    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache" pod "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" (UID: "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:43.319655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.319632    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:43.319768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.319658    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:43.319768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.319672    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:43.321825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.321796    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs" pod "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" (UID: "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:43.321986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.321819    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw" pod "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" (UID: "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3"). InnerVolumeSpecName "kube-api-access-l99jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:43.324357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.324331    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm" pod "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" (UID: "7bb87dcd-a9b6-464a-95da-a4e0f007e8a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:43.421250 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.421212    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:43.421250 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.421244    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l99jw\" (UniqueName: \"kubernetes.io/projected/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-kube-api-access-l99jw\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:43.421250 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.421256    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:43.946412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.946386    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:46:43.986698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.986623    4451 generic.go:356] "Generic (PLEG): container finished" podID="7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" containerID="22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220" exitCode=0
Oct 29 22:46:43.986698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.986671    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"
Oct 29 22:46:43.986928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.986707    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28" event={"ID":"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3","Type":"ContainerDied","Data":"22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220"}
Oct 29 22:46:43.986928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.986751    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28" event={"ID":"7bb87dcd-a9b6-464a-95da-a4e0f007e8a3","Type":"ContainerDied","Data":"65dbd6cfa612fd639d75623be6cfa0d7d67148809248102490c6881bc4494f25"}
Oct 29 22:46:43.986928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.986767    4451 scope.go:122] "RemoveContainer" containerID="22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220"
Oct 29 22:46:43.988308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.988280    4451 generic.go:356] "Generic (PLEG): container finished" podID="2212fa81-5872-42bf-9acd-831276cfec3e" containerID="2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148" exitCode=0
Oct 29 22:46:43.988683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.988344    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"
Oct 29 22:46:43.988683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.988370    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" event={"ID":"2212fa81-5872-42bf-9acd-831276cfec3e","Type":"ContainerDied","Data":"2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148"}
Oct 29 22:46:43.988683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.988424    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp" event={"ID":"2212fa81-5872-42bf-9acd-831276cfec3e","Type":"ContainerDied","Data":"424334aeb7d67eba6c3b02d2de39553b57d11808e9ea8906a55a3b2f3c4b56f8"}
Oct 29 22:46:43.999003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.998803    4451 scope.go:122] "RemoveContainer" containerID="22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220"
Oct 29 22:46:43.999235 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:46:43.999208    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220\": container with ID starting with 22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220 not found: ID does not exist" containerID="22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220"
Oct 29 22:46:43.999303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.999248    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220"} err="failed to get container status \"22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220\": rpc error: code = NotFound desc = could not find container \"22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220\": container with ID starting with 22442f7a5dd05bb498d86470aab2438f26e91fa04d915f57d7c5193ced181220 not found: ID does not exist"
Oct 29 22:46:43.999303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:43.999272    4451 scope.go:122] "RemoveContainer" containerID="2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148"
Oct 29 22:46:44.029636 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.029610    4451 scope.go:122] "RemoveContainer" containerID="2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148"
Oct 29 22:46:44.029782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.029760    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.029891 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.029875    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.029989    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030037    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030154 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:46:44.030063    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148\": container with ID starting with 2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148 not found: ID does not exist" containerID="2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148"
Oct 29 22:46:44.030154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030078    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030105    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030115    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148"} err="failed to get container status \"2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148\": rpc error: code = NotFound desc = could not find container \"2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148\": container with ID starting with 2c0973f0491e18a1492db9cff60962520e5108b8a64b6117b62eb352f82db148 not found: ID does not exist"
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030168    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030354    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030367    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030406    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data\") pod \"2212fa81-5872-42bf-9acd-831276cfec3e\" (UID: \"2212fa81-5872-42bf-9acd-831276cfec3e\") "
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030462    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030721    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.030886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.030826    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.031319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.031246    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.031319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.031271    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.031319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.031287    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2212fa81-5872-42bf-9acd-831276cfec3e-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.031319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.031305    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.031529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.031368    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.032847 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.032824    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "kube-api-access-22rpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:44.034165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.034143    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:46:44.034244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.034164    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:44.035405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.035384    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy" pod "2212fa81-5872-42bf-9acd-831276cfec3e" (UID: "2212fa81-5872-42bf-9acd-831276cfec3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.054377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.054313    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"]
Oct 29 22:46:44.060193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.060166    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8d985a/test-model-8d985a-kserve-5cb869c8cc-llw28"]
Oct 29 22:46:44.137498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.134313    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.137498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.134351    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.137498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.134365    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2212fa81-5872-42bf-9acd-831276cfec3e-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.137498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.134387    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rpz\" (UniqueName: \"kubernetes.io/projected/2212fa81-5872-42bf-9acd-831276cfec3e-kube-api-access-22rpz\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.137498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.134401    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2212fa81-5872-42bf-9acd-831276cfec3e-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.165242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.165212    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" path="/var/lib/kubelet/pods/7bb87dcd-a9b6-464a-95da-a4e0f007e8a3/volumes"
Oct 29 22:46:44.324126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.324094    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"]
Oct 29 22:46:44.333965 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:44.333939    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-rppfp"]
Oct 29 22:46:46.165365 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:46.165334    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" path="/var/lib/kubelet/pods/2212fa81-5872-42bf-9acd-831276cfec3e/volumes"
Oct 29 22:46:48.742186 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:46:48.742150    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:46:52.869332 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:52.869299    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"]
Oct 29 22:46:52.869783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:52.869520    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy" containerID="cri-o://a7c6eaf75092ca953cc9d9a59c0835ac2a1c47daea95adc5d244cca7ee8a766c" gracePeriod=30
Oct 29 22:46:56.389746 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.389710    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"]
Oct 29 22:46:56.390383 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.390368    4451 cpu_manager.go:414] "containerMap: removing container" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy"
Oct 29 22:46:56.390429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.390383    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy"
Oct 29 22:46:56.390429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.390410    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" containerName="main"
Oct 29 22:46:56.390429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.390417    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" containerName="main"
Oct 29 22:46:56.390546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.390536    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="2212fa81-5872-42bf-9acd-831276cfec3e" containerName="istio-proxy"
Oct 29 22:46:56.390582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.390552    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb87dcd-a9b6-464a-95da-a4e0f007e8a3" containerName="main"
Oct 29 22:46:56.394649 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.394628    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.411848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.411817    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"]
Oct 29 22:46:56.416399 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.416327    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-af260f-serving-cert\""
Oct 29 22:46:56.435240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.435204    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.435552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.435523    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.435668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.435647    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.435753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.435736    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6c65\" (UniqueName: \"kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.536599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.536562    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.536801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.536653    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.536801 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:46:56.536717    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-serving-cert: secret "maas-api-e2e-shared-a-w1-af260f-serving-cert" not found
Oct 29 22:46:56.536801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.536720    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.536801 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:46:56.536797    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls podName:7659018a-3ad3-4808-a8bc-857765012d06 nodeName:}" failed. No retries permitted until 2026-10-29 22:46:57.036780566 +0000 UTC m=+1956.979739262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls") pod "maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" (UID: "7659018a-3ad3-4808-a8bc-857765012d06") : secret "maas-api-e2e-shared-a-w1-af260f-serving-cert" not found
Oct 29 22:46:56.537079 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.536894    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6c65\" (UniqueName: \"kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.537437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.537416    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.539176 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.539155    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:56.554210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:56.554183    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6c65\" (UniqueName: \"kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:57.044790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:57.044745    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:57.047241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:57.047217    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:57.310610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:57.310528    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:57.635798 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:46:57.635764    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7659018a_3ad3_4808_a8bc_857765012d06.slice/crio-dd1922253406f8dd198e02bca39f0122d4728b39597b09da2eaf7fdb88d6f96b WatchSource:0}: Error finding container dd1922253406f8dd198e02bca39f0122d4728b39597b09da2eaf7fdb88d6f96b: Status 404 returned error can't find the container with id dd1922253406f8dd198e02bca39f0122d4728b39597b09da2eaf7fdb88d6f96b
Oct 29 22:46:57.636423 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:57.636262    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"]
Oct 29 22:46:58.137771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.137729    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" event={"ID":"7659018a-3ad3-4808-a8bc-857765012d06","Type":"ContainerStarted","Data":"4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e"}
Oct 29 22:46:58.137956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.137806    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:46:58.137956 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.137828    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" event={"ID":"7659018a-3ad3-4808-a8bc-857765012d06","Type":"ContainerStarted","Data":"dd1922253406f8dd198e02bca39f0122d4728b39597b09da2eaf7fdb88d6f96b"}
Oct 29 22:46:58.139372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.139348    4451 generic.go:356] "Generic (PLEG): container finished" podID="37056749-6ab5-4d80-8025-b0e32e11601f" containerID="a7c6eaf75092ca953cc9d9a59c0835ac2a1c47daea95adc5d244cca7ee8a766c" exitCode=0
Oct 29 22:46:58.139481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.139420    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" event={"ID":"37056749-6ab5-4d80-8025-b0e32e11601f","Type":"ContainerDied","Data":"a7c6eaf75092ca953cc9d9a59c0835ac2a1c47daea95adc5d244cca7ee8a766c"}
Oct 29 22:46:58.139481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.139461    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm" event={"ID":"37056749-6ab5-4d80-8025-b0e32e11601f","Type":"ContainerDied","Data":"ffb440b7aeb3092de61a0fa481403c2394053815df81fde0be6be0e3d28fa8b6"}
Oct 29 22:46:58.139481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.139479    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb440b7aeb3092de61a0fa481403c2394053815df81fde0be6be0e3d28fa8b6"
Oct 29 22:46:58.160522 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.160493    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:46:58.238257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.238193    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" podStartSLOduration=1.90775808 podStartE2EDuration="2.23816951s" totalImagesPullingTime="330.41143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:58.196430318 +0000 UTC m=+1958.139389024" watchObservedRunningTime="2026-10-29 22:46:58.23816951 +0000 UTC m=+1958.181128215"
Oct 29 22:46:58.300052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300013    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300066    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300103    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300168    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300291    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:58.300323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300296    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300384    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300406    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.300536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300431    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.301485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300641    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data\") pod \"37056749-6ab5-4d80-8025-b0e32e11601f\" (UID: \"37056749-6ab5-4d80-8025-b0e32e11601f\") "
Oct 29 22:46:58.301485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300645    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:58.301485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.300713    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:58.301485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.301013    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:58.301485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.301120    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:58.302005 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.301982    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.302096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.302018    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.302096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.302040    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37056749-6ab5-4d80-8025-b0e32e11601f-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.302096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.302056    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.302096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.302071    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.302803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.302782    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:58.303130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.303106    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:58.303216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.303166    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:46:58.303323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.303307    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd" pod "37056749-6ab5-4d80-8025-b0e32e11601f" (UID: "37056749-6ab5-4d80-8025-b0e32e11601f"). InnerVolumeSpecName "kube-api-access-4wxpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:58.403654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.403558    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.403654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.403591    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37056749-6ab5-4d80-8025-b0e32e11601f-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.403654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.403604    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxpd\" (UniqueName: \"kubernetes.io/projected/37056749-6ab5-4d80-8025-b0e32e11601f-kube-api-access-4wxpd\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:58.403654 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:58.403618    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37056749-6ab5-4d80-8025-b0e32e11601f-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:46:59.146020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:59.145983    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"
Oct 29 22:46:59.205664 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:59.205627    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"]
Oct 29 22:46:59.216984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:46:59.216957    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-fqhkm"]
Oct 29 22:47:00.164736 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:47:00.164703    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" path="/var/lib/kubelet/pods/37056749-6ab5-4d80-8025-b0e32e11601f/volumes"
Oct 29 22:47:04.152111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:47:04.152076    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 22:47:18.745144 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:47:18.745102    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:47:20.892985 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:47:20.892955    4451 scope.go:122] "RemoveContainer" containerID="a7c6eaf75092ca953cc9d9a59c0835ac2a1c47daea95adc5d244cca7ee8a766c"
Oct 29 22:47:47.260083 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:47:47.249374    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"]
Oct 29 22:47:48.742819 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:47:48.742776    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:48:02.458904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.458830    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" podUID="b28aef5a-c6f0-415c-913c-35c8085afde6" containerName="main" containerID="cri-o://bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72" gracePeriod=45
Oct 29 22:48:02.742256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.742232    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:48:02.785189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785156    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home\") pod \"b28aef5a-c6f0-415c-913c-35c8085afde6\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") "
Oct 29 22:48:02.785360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785208    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4\" (UniqueName: \"kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4\") pod \"b28aef5a-c6f0-415c-913c-35c8085afde6\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") "
Oct 29 22:48:02.785360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785297    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir\") pod \"b28aef5a-c6f0-415c-913c-35c8085afde6\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") "
Oct 29 22:48:02.785476 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785409    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache\") pod \"b28aef5a-c6f0-415c-913c-35c8085afde6\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") "
Oct 29 22:48:02.785476 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785445    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home" pod "b28aef5a-c6f0-415c-913c-35c8085afde6" (UID: "b28aef5a-c6f0-415c-913c-35c8085afde6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:02.785567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785525    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir" pod "b28aef5a-c6f0-415c-913c-35c8085afde6" (UID: "b28aef5a-c6f0-415c-913c-35c8085afde6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:02.785567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785537    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm\") pod \"b28aef5a-c6f0-415c-913c-35c8085afde6\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") "
Oct 29 22:48:02.785655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785639    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache" pod "b28aef5a-c6f0-415c-913c-35c8085afde6" (UID: "b28aef5a-c6f0-415c-913c-35c8085afde6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:02.785706 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.785683    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs\" (UniqueName: \"kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs\") pod \"b28aef5a-c6f0-415c-913c-35c8085afde6\" (UID: \"b28aef5a-c6f0-415c-913c-35c8085afde6\") "
Oct 29 22:48:02.786441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.786414    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.786584 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.786466    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.786584 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.786480    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.788211 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.788183    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs" pod "b28aef5a-c6f0-415c-913c-35c8085afde6" (UID: "b28aef5a-c6f0-415c-913c-35c8085afde6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:02.788419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.788400    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4" pod "b28aef5a-c6f0-415c-913c-35c8085afde6" (UID: "b28aef5a-c6f0-415c-913c-35c8085afde6"). InnerVolumeSpecName "kube-api-access-8c2g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:02.789055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.789026    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm" pod "b28aef5a-c6f0-415c-913c-35c8085afde6" (UID: "b28aef5a-c6f0-415c-913c-35c8085afde6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:02.798386 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.798356    4451 generic.go:356] "Generic (PLEG): container finished" podID="b28aef5a-c6f0-415c-913c-35c8085afde6" containerID="bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72" exitCode=0
Oct 29 22:48:02.798498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.798402    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"
Oct 29 22:48:02.798603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.798582    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" event={"ID":"b28aef5a-c6f0-415c-913c-35c8085afde6","Type":"ContainerDied","Data":"bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72"}
Oct 29 22:48:02.798789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.798620    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" event={"ID":"b28aef5a-c6f0-415c-913c-35c8085afde6","Type":"ContainerDied","Data":"4aaefe6d4b28dc53543a8afa94f7c77d04969ac6c2030a2818b6f8e800c8ac1f"}
Oct 29 22:48:02.798789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.798648    4451 scope.go:122] "RemoveContainer" containerID="bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72"
Oct 29 22:48:02.825435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.825398    4451 scope.go:122] "RemoveContainer" containerID="bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72"
Oct 29 22:48:02.825866 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:48:02.825836    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72\": container with ID starting with bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72 not found: ID does not exist" containerID="bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72"
Oct 29 22:48:02.826001 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.825876    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72"} err="failed to get container status \"bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72\": rpc error: code = NotFound desc = could not find container \"bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72\": container with ID starting with bd4206c9f5b6ec69e23402d53b5c9fc7e79242e9cd52f407f7840eca8f6c9b72 not found: ID does not exist"
Oct 29 22:48:02.854411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.854385    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"]
Oct 29 22:48:02.857723 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.857699    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d"]
Oct 29 22:48:02.887757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.887723    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28aef5a-c6f0-415c-913c-35c8085afde6-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.887757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.887759    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28aef5a-c6f0-415c-913c-35c8085afde6-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:02.887897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:02.887771    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c2g4\" (UniqueName: \"kubernetes.io/projected/b28aef5a-c6f0-415c-913c-35c8085afde6-kube-api-access-8c2g4\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:03.683514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:03.683476    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fee294/test-model-fee294-kserve-5969ccbf7-sqw4d" podUID="b28aef5a-c6f0-415c-913c-35c8085afde6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:48:04.166493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:04.166457    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28aef5a-c6f0-415c-913c-35c8085afde6" path="/var/lib/kubelet/pods/b28aef5a-c6f0-415c-913c-35c8085afde6/volumes"
Oct 29 22:48:18.742123 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:48:18.742085    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:48:24.873979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:24.873938    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"]
Oct 29 22:48:24.874481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:24.874197    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" podUID="d46a5f45-8d76-46db-b2d6-9055f4e13071" containerName="discovery" containerID="cri-o://8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98" gracePeriod=30
Oct 29 22:48:25.166754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.166726    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:48:25.233482 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.233446    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.235355 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.234899    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.235905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.235872    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.236171 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.236151    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.236675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.235530    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:25.236854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.236836    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.237424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.237406    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.237894 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.237876    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q\") pod \"d46a5f45-8d76-46db-b2d6-9055f4e13071\" (UID: \"d46a5f45-8d76-46db-b2d6-9055f4e13071\") "
Oct 29 22:48:25.238986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.238965    4451 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-ca-configmap\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.241689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.241661    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "kube-api-access-28k8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:25.242902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.242857    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:25.242902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.242866    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:25.243093 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.243051    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:25.244146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.244108    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:25.244463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.244439    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts" pod "d46a5f45-8d76-46db-b2d6-9055f4e13071" (UID: "d46a5f45-8d76-46db-b2d6-9055f4e13071"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:25.340251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.340217    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.340251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.340246    4451 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-cacerts\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.340251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.340255    4451 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a5f45-8d76-46db-b2d6-9055f4e13071-local-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.340494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.340263    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28k8q\" (UniqueName: \"kubernetes.io/projected/d46a5f45-8d76-46db-b2d6-9055f4e13071-kube-api-access-28k8q\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.340494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.340277    4451 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-csr-dns-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.340494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.340296    4451 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d46a5f45-8d76-46db-b2d6-9055f4e13071-istio-kubeconfig\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:48:25.969773 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.969729    4451 generic.go:356] "Generic (PLEG): container finished" podID="d46a5f45-8d76-46db-b2d6-9055f4e13071" containerID="8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98" exitCode=0
Oct 29 22:48:25.970292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.969815    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" event={"ID":"d46a5f45-8d76-46db-b2d6-9055f4e13071","Type":"ContainerDied","Data":"8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98"}
Oct 29 22:48:25.970292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.969864    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm" event={"ID":"d46a5f45-8d76-46db-b2d6-9055f4e13071","Type":"ContainerDied","Data":"d65dc734d43b0222a1ac6ca8137cae6d5664cc19fb4bfc553129b3420a4ac15f"}
Oct 29 22:48:25.970292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.969890    4451 scope.go:122] "RemoveContainer" containerID="8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98"
Oct 29 22:48:25.970292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:25.970062    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"
Oct 29 22:48:26.002716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:26.002682    4451 scope.go:122] "RemoveContainer" containerID="8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98"
Oct 29 22:48:26.003127 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:48:26.003101    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98\": container with ID starting with 8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98 not found: ID does not exist" containerID="8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98"
Oct 29 22:48:26.003214 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:26.003138    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98"} err="failed to get container status \"8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98\": rpc error: code = NotFound desc = could not find container \"8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98\": container with ID starting with 8a55022e9a6ebb0429d88dde81f9b9737bafecd8d01232c9c077e8f351e82a98 not found: ID does not exist"
Oct 29 22:48:26.041410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:26.041375    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"]
Oct 29 22:48:26.051461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:26.051432    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-267vm"]
Oct 29 22:48:26.179007 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:26.178668    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46a5f45-8d76-46db-b2d6-9055f4e13071" path="/var/lib/kubelet/pods/d46a5f45-8d76-46db-b2d6-9055f4e13071/volumes"
Oct 29 22:48:39.958151 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958114    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"]
Oct 29 22:48:39.958800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958784    4451 cpu_manager.go:414] "containerMap: removing container" podUID="d46a5f45-8d76-46db-b2d6-9055f4e13071" containerName="discovery"
Oct 29 22:48:39.958853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958802    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46a5f45-8d76-46db-b2d6-9055f4e13071" containerName="discovery"
Oct 29 22:48:39.958853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958817    4451 cpu_manager.go:414] "containerMap: removing container" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy"
Oct 29 22:48:39.958853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958822    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy"
Oct 29 22:48:39.958853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958843    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b28aef5a-c6f0-415c-913c-35c8085afde6" containerName="main"
Oct 29 22:48:39.958853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958848    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28aef5a-c6f0-415c-913c-35c8085afde6" containerName="main"
Oct 29 22:48:39.959025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958966    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46a5f45-8d76-46db-b2d6-9055f4e13071" containerName="discovery"
Oct 29 22:48:39.959025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958975    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="37056749-6ab5-4d80-8025-b0e32e11601f" containerName="istio-proxy"
Oct 29 22:48:39.959025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.958985    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28aef5a-c6f0-415c-913c-35c8085afde6" containerName="main"
Oct 29 22:48:39.963893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.963874    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:39.980514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.980491    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"]
Oct 29 22:48:39.986046 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:39.986021    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f8ca96-serving-cert\""
Oct 29 22:48:40.062360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.062303    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mp2\" (UniqueName: \"kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.062617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.062541    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.062697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.062662    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.062837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.062779    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.163819 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.163787    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mp2\" (UniqueName: \"kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.164049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.163959    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.164118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.164047    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.164179 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.164160    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.165061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.165031    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.166767 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.166744    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.167313 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.167292    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.192045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.192011    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mp2\" (UniqueName: \"kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2\") pod \"maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.279638 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.279554    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:40.586494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.586463    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"]
Oct 29 22:48:40.586845 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:48:40.586823    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc86164_b7a8_447d_8656_dc6016c4a134.slice/crio-fb25fb0c5ab17b5461dbae23c7acbf75b1ea3399a61709721ef3722dd5578dda WatchSource:0}: Error finding container fb25fb0c5ab17b5461dbae23c7acbf75b1ea3399a61709721ef3722dd5578dda: Status 404 returned error can't find the container with id fb25fb0c5ab17b5461dbae23c7acbf75b1ea3399a61709721ef3722dd5578dda
Oct 29 22:48:40.588430 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:40.588413    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:48:41.110484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:41.110448    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6" event={"ID":"1fc86164-b7a8-447d-8656-dc6016c4a134","Type":"ContainerStarted","Data":"99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999"}
Oct 29 22:48:41.110879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:41.110506    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6" event={"ID":"1fc86164-b7a8-447d-8656-dc6016c4a134","Type":"ContainerStarted","Data":"fb25fb0c5ab17b5461dbae23c7acbf75b1ea3399a61709721ef3722dd5578dda"}
Oct 29 22:48:41.110879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:41.110535    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:41.152481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:41.152408    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6" podStartSLOduration=1.765804396 podStartE2EDuration="2.152387145s" totalImagesPullingTime="386.582749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:41.145887907 +0000 UTC m=+2061.088846613" watchObservedRunningTime="2026-10-29 22:48:41.152387145 +0000 UTC m=+2061.095345851"
Oct 29 22:48:47.123519 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:47.123479    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:48:48.744690 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:48:48.744652    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:48:56.338936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:48:56.338092    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"]
Oct 29 22:49:02.836935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:02.834057    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"]
Oct 29 22:49:04.894364 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:04.894322    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"]
Oct 29 22:49:04.895178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:04.894605    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" podUID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" containerName="maas-api" containerID="cri-o://dead648187c56ed868a26300b9d60abf935cb820b7383ed7a324a56e58465da4" gracePeriod=30
Oct 29 22:49:05.482451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.482199    4451 generic.go:356] "Generic (PLEG): container finished" podID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" containerID="dead648187c56ed868a26300b9d60abf935cb820b7383ed7a324a56e58465da4" exitCode=0
Oct 29 22:49:05.482451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.482339    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" event={"ID":"f4ccd5cb-61b6-4f5e-92da-787746ee51d5","Type":"ContainerDied","Data":"dead648187c56ed868a26300b9d60abf935cb820b7383ed7a324a56e58465da4"}
Oct 29 22:49:05.603567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.603540    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:49:05.685595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.685557    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca\") pod \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") "
Oct 29 22:49:05.685797 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.685618    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs\") pod \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") "
Oct 29 22:49:05.685871 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.685829    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls\") pod \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") "
Oct 29 22:49:05.685951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.685879    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp\" (UniqueName: \"kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp\") pod \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\" (UID: \"f4ccd5cb-61b6-4f5e-92da-787746ee51d5\") "
Oct 29 22:49:05.686064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.686028    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca" pod "f4ccd5cb-61b6-4f5e-92da-787746ee51d5" (UID: "f4ccd5cb-61b6-4f5e-92da-787746ee51d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:05.686598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.686569    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:05.694811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.694779    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls" pod "f4ccd5cb-61b6-4f5e-92da-787746ee51d5" (UID: "f4ccd5cb-61b6-4f5e-92da-787746ee51d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:05.694811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.694802    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs" pod "f4ccd5cb-61b6-4f5e-92da-787746ee51d5" (UID: "f4ccd5cb-61b6-4f5e-92da-787746ee51d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:05.695025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.694817    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp" pod "f4ccd5cb-61b6-4f5e-92da-787746ee51d5" (UID: "f4ccd5cb-61b6-4f5e-92da-787746ee51d5"). InnerVolumeSpecName "kube-api-access-h9lwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:05.788783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.788702    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:05.788783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.788733    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9lwp\" (UniqueName: \"kubernetes.io/projected/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-kube-api-access-h9lwp\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:05.788783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:05.788746    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ccd5cb-61b6-4f5e-92da-787746ee51d5-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:06.489784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:06.489708    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"
Oct 29 22:49:06.490253 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:06.489711    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" event={"ID":"f4ccd5cb-61b6-4f5e-92da-787746ee51d5","Type":"ContainerDied","Data":"e7701c45883b2ea5821f6c1f9c035d779d676ed74b718b7405c2d331e3c6ee99"}
Oct 29 22:49:06.490253 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:06.489839    4451 scope.go:122] "RemoveContainer" containerID="dead648187c56ed868a26300b9d60abf935cb820b7383ed7a324a56e58465da4"
Oct 29 22:49:06.535569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:06.535541    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"]
Oct 29 22:49:06.546243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:06.546217    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp"]
Oct 29 22:49:08.165447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:08.165408    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" path="/var/lib/kubelet/pods/f4ccd5cb-61b6-4f5e-92da-787746ee51d5/volumes"
Oct 29 22:49:08.550966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:08.550840    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5af18-689b56fbc7-w8knp" podUID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.90:8443/health\": context deadline exceeded"
Oct 29 22:49:08.763168 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:08.763132    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"]
Oct 29 22:49:08.763467 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:08.763435    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" podUID="e30f6550-cc1b-49a3-8a87-a5d2d02317b8" containerName="maas-api" containerID="cri-o://d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501" gracePeriod=30
Oct 29 22:49:09.069924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.069893    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:49:09.143973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.143871    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\") pod \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") "
Oct 29 22:49:09.144124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.144025    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb\" (UniqueName: \"kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb\") pod \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") "
Oct 29 22:49:09.144124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.144060    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs\") pod \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") "
Oct 29 22:49:09.144124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.144085    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca\") pod \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\" (UID: \"e30f6550-cc1b-49a3-8a87-a5d2d02317b8\") "
Oct 29 22:49:09.144835 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.144805    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca" pod "e30f6550-cc1b-49a3-8a87-a5d2d02317b8" (UID: "e30f6550-cc1b-49a3-8a87-a5d2d02317b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:09.146688 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.146651    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb" pod "e30f6550-cc1b-49a3-8a87-a5d2d02317b8" (UID: "e30f6550-cc1b-49a3-8a87-a5d2d02317b8"). InnerVolumeSpecName "kube-api-access-4lzkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:09.146796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.146765    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls" pod "e30f6550-cc1b-49a3-8a87-a5d2d02317b8" (UID: "e30f6550-cc1b-49a3-8a87-a5d2d02317b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:09.147096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.146905    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs" pod "e30f6550-cc1b-49a3-8a87-a5d2d02317b8" (UID: "e30f6550-cc1b-49a3-8a87-a5d2d02317b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:09.245571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.245531    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:09.245571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.245565    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzkb\" (UniqueName: \"kubernetes.io/projected/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-kube-api-access-4lzkb\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:09.246125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.245579    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:09.246125 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.245591    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e30f6550-cc1b-49a3-8a87-a5d2d02317b8-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:09.513074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.512985    4451 generic.go:356] "Generic (PLEG): container finished" podID="e30f6550-cc1b-49a3-8a87-a5d2d02317b8" containerID="d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501" exitCode=0
Oct 29 22:49:09.513074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.513048    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"
Oct 29 22:49:09.513287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.513076    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" event={"ID":"e30f6550-cc1b-49a3-8a87-a5d2d02317b8","Type":"ContainerDied","Data":"d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501"}
Oct 29 22:49:09.513287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.513133    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln" event={"ID":"e30f6550-cc1b-49a3-8a87-a5d2d02317b8","Type":"ContainerDied","Data":"88062ecb3a85c1f6b06d2ccb11d3a691576a7660af721adb19a002be5878ae00"}
Oct 29 22:49:09.513287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.513157    4451 scope.go:122] "RemoveContainer" containerID="d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501"
Oct 29 22:49:09.525078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.525058    4451 scope.go:122] "RemoveContainer" containerID="d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501"
Oct 29 22:49:09.525467 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:09.525442    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501\": container with ID starting with d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501 not found: ID does not exist" containerID="d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501"
Oct 29 22:49:09.525599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.525486    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501"} err="failed to get container status \"d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501\": rpc error: code = NotFound desc = could not find container \"d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501\": container with ID starting with d06d267e88692b67a0cc0fafe41940b50d16c2f5bcffe625208d51c059f11501 not found: ID does not exist"
Oct 29 22:49:09.578575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.578338    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"]
Oct 29 22:49:09.584391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:09.584351    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7d192-5f744466ff-jc8ln"]
Oct 29 22:49:10.166107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.166076    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30f6550-cc1b-49a3-8a87-a5d2d02317b8" path="/var/lib/kubelet/pods/e30f6550-cc1b-49a3-8a87-a5d2d02317b8/volumes"
Oct 29 22:49:10.166460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.166444    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"]
Oct 29 22:49:10.167052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.167038    4451 cpu_manager.go:414] "containerMap: removing container" podUID="e30f6550-cc1b-49a3-8a87-a5d2d02317b8" containerName="maas-api"
Oct 29 22:49:10.167110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.167053    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30f6550-cc1b-49a3-8a87-a5d2d02317b8" containerName="maas-api"
Oct 29 22:49:10.167110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.167066    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" containerName="maas-api"
Oct 29 22:49:10.167110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.167071    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" containerName="maas-api"
Oct 29 22:49:10.167208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.167154    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30f6550-cc1b-49a3-8a87-a5d2d02317b8" containerName="maas-api"
Oct 29 22:49:10.167208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.167165    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ccd5cb-61b6-4f5e-92da-787746ee51d5" containerName="maas-api"
Oct 29 22:49:10.172167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.172141    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.188759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.188727    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"]
Oct 29 22:49:10.261754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.261719    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.261754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.261775    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnct\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.261845    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.262002    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.262080    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.262137    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.262189    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.262312    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.262359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.262353    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.282348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.282324    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e619b6-openshift-default-dockercfg-rrqdd\""
Oct 29 22:49:10.363656 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.363631    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.363796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.363689    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.363796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.363736    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.363796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.363767    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.363989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.363858    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.363989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.363898    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364039    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364081    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnct\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364082    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364105    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364109    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364386 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364139    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364418    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.364936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.364885    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.366356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.366331    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.366650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.366630    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.381799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.381773    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.381997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.381981    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnct\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.487815 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.487730    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:10.807420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.807392    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"]
Oct 29 22:49:10.807794 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:49:10.807773    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14413f3_c6ae_4f73_8bcd_bedc843cfba7.slice/crio-5c38a900ea61d00d5bc591bc22a9886ebde1bf3ea134453ce92c5639b48a7b2f WatchSource:0}: Error finding container 5c38a900ea61d00d5bc591bc22a9886ebde1bf3ea134453ce92c5639b48a7b2f: Status 404 returned error can't find the container with id 5c38a900ea61d00d5bc591bc22a9886ebde1bf3ea134453ce92c5639b48a7b2f
Oct 29 22:49:10.810394 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.810363    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:49:10.810481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.810426    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:49:10.810481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:10.810454    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:49:11.447990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.440065    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292" podUID="b2c68168-f92c-4425-a8d5-843d540ebb90" containerName="main" containerID="cri-o://5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6" gracePeriod=45
Oct 29 22:49:11.532652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.532604    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" event={"ID":"b14413f3-c6ae-4f73-8bcd-bedc843cfba7","Type":"ContainerStarted","Data":"552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1"}
Oct 29 22:49:11.532837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.532683    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" event={"ID":"b14413f3-c6ae-4f73-8bcd-bedc843cfba7","Type":"ContainerStarted","Data":"5c38a900ea61d00d5bc591bc22a9886ebde1bf3ea134453ce92c5639b48a7b2f"}
Oct 29 22:49:11.588765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.588708    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podStartSLOduration=1.5886931180000001 podStartE2EDuration="1.588693118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:11.585527028 +0000 UTC m=+2091.528485732" watchObservedRunningTime="2026-10-29 22:49:11.588693118 +0000 UTC m=+2091.531651823"
Oct 29 22:49:11.788417 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.788393    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.936748    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home\") pod \"b2c68168-f92c-4425-a8d5-843d540ebb90\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") "
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.936845    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache\") pod \"b2c68168-f92c-4425-a8d5-843d540ebb90\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") "
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937011    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm\") pod \"b2c68168-f92c-4425-a8d5-843d540ebb90\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") "
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937059    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir\") pod \"b2c68168-f92c-4425-a8d5-843d540ebb90\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") "
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937116    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl\" (UniqueName: \"kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl\") pod \"b2c68168-f92c-4425-a8d5-843d540ebb90\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") "
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937162    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home" pod "b2c68168-f92c-4425-a8d5-843d540ebb90" (UID: "b2c68168-f92c-4425-a8d5-843d540ebb90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937222    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs\") pod \"b2c68168-f92c-4425-a8d5-843d540ebb90\" (UID: \"b2c68168-f92c-4425-a8d5-843d540ebb90\") "
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937402    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir" pod "b2c68168-f92c-4425-a8d5-843d540ebb90" (UID: "b2c68168-f92c-4425-a8d5-843d540ebb90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937828    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.937846    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:11.938967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.938097    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache" pod "b2c68168-f92c-4425-a8d5-843d540ebb90" (UID: "b2c68168-f92c-4425-a8d5-843d540ebb90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:11.940227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.940161    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm" pod "b2c68168-f92c-4425-a8d5-843d540ebb90" (UID: "b2c68168-f92c-4425-a8d5-843d540ebb90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:11.940227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.940187    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl" pod "b2c68168-f92c-4425-a8d5-843d540ebb90" (UID: "b2c68168-f92c-4425-a8d5-843d540ebb90"). InnerVolumeSpecName "kube-api-access-v89dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:11.941301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:11.941265    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs" pod "b2c68168-f92c-4425-a8d5-843d540ebb90" (UID: "b2c68168-f92c-4425-a8d5-843d540ebb90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:12.039690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.039606    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:12.039690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.039644    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89dl\" (UniqueName: \"kubernetes.io/projected/b2c68168-f92c-4425-a8d5-843d540ebb90-kube-api-access-v89dl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:12.039690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.039661    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c68168-f92c-4425-a8d5-843d540ebb90-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:12.039690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.039675    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c68168-f92c-4425-a8d5-843d540ebb90-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:12.488327 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.488294    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:12.517028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.516089    4451 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:49:12.517028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.516275    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:49:12.541339 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.541308    4451 generic.go:356] "Generic (PLEG): container finished" podID="b2c68168-f92c-4425-a8d5-843d540ebb90" containerID="5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6" exitCode=0
Oct 29 22:49:12.541534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.541354    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"
Oct 29 22:49:12.541534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.541393    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292" event={"ID":"b2c68168-f92c-4425-a8d5-843d540ebb90","Type":"ContainerDied","Data":"5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6"}
Oct 29 22:49:12.541534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.541443    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292" event={"ID":"b2c68168-f92c-4425-a8d5-843d540ebb90","Type":"ContainerDied","Data":"ef49120e8bebc95eacf2590027f9a8a065fceb85e63585b206dd24a78fef2520"}
Oct 29 22:49:12.541534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.541465    4451 scope.go:122] "RemoveContainer" containerID="5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6"
Oct 29 22:49:12.551429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.551405    4451 scope.go:122] "RemoveContainer" containerID="5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6"
Oct 29 22:49:12.551702 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:12.551684    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6\": container with ID starting with 5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6 not found: ID does not exist" containerID="5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6"
Oct 29 22:49:12.551756 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.551711    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6"} err="failed to get container status \"5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6\": rpc error: code = NotFound desc = could not find container \"5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6\": container with ID starting with 5871e9f75b26dc9ba69e36d5596bc63e3062f331e63ec8bcfd531674f42893e6 not found: ID does not exist"
Oct 29 22:49:12.588265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.588236    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"]
Oct 29 22:49:12.613246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:12.610689    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2a3ef0/test-model-2a3ef0-kserve-7d4cc8cff7-q2292"]
Oct 29 22:49:13.488478 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:13.488443    4451 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:49:13.489021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:13.488522    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:49:14.166513 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:14.166470    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c68168-f92c-4425-a8d5-843d540ebb90" path="/var/lib/kubelet/pods/b2c68168-f92c-4425-a8d5-843d540ebb90/volumes"
Oct 29 22:49:14.488509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:14.488431    4451 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:49:14.488509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:14.488491    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:49:15.488456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:15.488424    4451 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:49:15.488669 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:15.488519    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:49:16.485800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:16.485761    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"]
Oct 29 22:49:16.486333 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:16.486289    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" containerID="cri-o://0c0390d9668f1b1ea482e5c08cd26876675d2638e652e7669ec6013c4a7e0a3a" gracePeriod=30
Oct 29 22:49:16.488692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:16.488661    4451 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:49:16.489122 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:16.488709    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:49:16.533486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:16.533454    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"]
Oct 29 22:49:16.533973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:16.533946    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" containerID="cri-o://d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27" gracePeriod=30
Oct 29 22:49:17.511363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:17.511329    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:17.512183 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:17.512162    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:17.513587 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:17.513564    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:17.862290 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:17.862253    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh" podUID="e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" containerName="main" containerID="cri-o://2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3" gracePeriod=45
Oct 29 22:49:18.312289 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.308578    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:49:18.408008 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.407971    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs\") pod \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") "
Oct 29 22:49:18.408177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.408057    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home\") pod \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") "
Oct 29 22:49:18.408177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.408139    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm\") pod \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") "
Oct 29 22:49:18.408315 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.408249    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir\") pod \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") "
Oct 29 22:49:18.408373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.408329    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache\") pod \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") "
Oct 29 22:49:18.408421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.408375    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh\" (UniqueName: \"kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh\") pod \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\" (UID: \"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b\") "
Oct 29 22:49:18.409243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.408755    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache" pod "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" (UID: "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.409243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.409057    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.409446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.409423    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home" pod "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" (UID: "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.409622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.409604    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir" pod "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" (UID: "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.411247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.411223    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh" pod "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" (UID: "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b"). InnerVolumeSpecName "kube-api-access-qbwdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:18.411778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.411753    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs" pod "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" (UID: "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:18.413448 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.413426    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm" pod "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" (UID: "e3ac0379-0e59-4ea2-a8b0-3a1cd844441b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.510995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.510952    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.510995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.510981    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.510995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.510996    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbwdh\" (UniqueName: \"kubernetes.io/projected/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-kube-api-access-qbwdh\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.511232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.511011    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.511232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.511026    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.589995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.589961    4451 generic.go:356] "Generic (PLEG): container finished" podID="e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" containerID="2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3" exitCode=0
Oct 29 22:49:18.590514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.590022    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"
Oct 29 22:49:18.590514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.590065    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh" event={"ID":"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b","Type":"ContainerDied","Data":"2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3"}
Oct 29 22:49:18.590514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.590113    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh" event={"ID":"e3ac0379-0e59-4ea2-a8b0-3a1cd844441b","Type":"ContainerDied","Data":"3c86966e1d71f941992d3b885fec56d09053c2cf0b530d8244cd1f6af556e320"}
Oct 29 22:49:18.590514 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.590135    4451 scope.go:122] "RemoveContainer" containerID="2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3"
Oct 29 22:49:18.605228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.605200    4451 scope.go:122] "RemoveContainer" containerID="2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3"
Oct 29 22:49:18.605583 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:18.605558    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3\": container with ID starting with 2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3 not found: ID does not exist" containerID="2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3"
Oct 29 22:49:18.605671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.605594    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3"} err="failed to get container status \"2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3\": rpc error: code = NotFound desc = could not find container \"2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3\": container with ID starting with 2d325e2a0da3c56f509c90852b3dbf361dfee4f21460e744b44ac2a2e2b6b3f3 not found: ID does not exist"
Oct 29 22:49:18.638892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.638860    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"]
Oct 29 22:49:18.642254 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:18.642228    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/auth-test-model-9aafe6-kserve-56db74b869-bppzh"]
Oct 29 22:49:18.751799 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:18.751684    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:49:19.683890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:19.683858    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:19.684274 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:19.683941    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:20.032798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:20.032716    4451 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:20.032959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:20.032802    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:20.164882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:20.164843    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" path="/var/lib/kubelet/pods/e3ac0379-0e59-4ea2-a8b0-3a1cd844441b/volumes"
Oct 29 22:49:21.621494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.621456    4451 generic.go:356] "Generic (PLEG): container finished" podID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerID="0c0390d9668f1b1ea482e5c08cd26876675d2638e652e7669ec6013c4a7e0a3a" exitCode=0
Oct 29 22:49:21.621854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.621563    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" event={"ID":"49c089e2-11bb-486d-a9e2-1eb04faa066f","Type":"ContainerDied","Data":"0c0390d9668f1b1ea482e5c08cd26876675d2638e652e7669ec6013c4a7e0a3a"}
Oct 29 22:49:21.704801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.704773    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904234    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904318    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904349    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904421    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904484    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904517    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904615    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904704    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.914021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.904782    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket\") pod \"49c089e2-11bb-486d-a9e2-1eb04faa066f\" (UID: \"49c089e2-11bb-486d-a9e2-1eb04faa066f\") "
Oct 29 22:49:21.921949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.920058    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:21.921949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.920439    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:21.921949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.920666    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:21.922394 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.922242    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:21.938935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.934108    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:21.940315 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.940263    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:21.955500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.955103    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:21.960496 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.960367    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "kube-api-access-v49mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:21.962720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.962681    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo" pod "49c089e2-11bb-486d-a9e2-1eb04faa066f" (UID: "49c089e2-11bb-486d-a9e2-1eb04faa066f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:21.965248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:21.965225    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:49:22.007585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007526    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.007585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007569    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.007585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007583    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.008020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007595    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.008020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007608    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.008020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007625    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.008020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007637    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c089e2-11bb-486d-a9e2-1eb04faa066f-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.008020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007648    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c089e2-11bb-486d-a9e2-1eb04faa066f-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.008020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.007660    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49mz\" (UniqueName: \"kubernetes.io/projected/49c089e2-11bb-486d-a9e2-1eb04faa066f-kube-api-access-v49mz\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.108062 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108035    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108091    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108126    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108247 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108184    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108241    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108301    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108353    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.108444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108438    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108485    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108553    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket\") pod \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\" (UID: \"6adcf1d7-002a-4c06-9f78-0b5af2347bcf\") "
Oct 29 22:49:22.108650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108579    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.108650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108635    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:22.108859 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108735    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.108978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.108958    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.109313 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.109291    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.109422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.109320    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.109422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.109336    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.109422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.109351    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.109422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.109364    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.110661 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.110627    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.110763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.110664    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "kube-api-access-sf2gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.110763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.110686    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:22.111196 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.111180    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token" pod "6adcf1d7-002a-4c06-9f78-0b5af2347bcf" (UID: "6adcf1d7-002a-4c06-9f78-0b5af2347bcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.211227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.211196    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.211227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.211228    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.211393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.211240    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.211393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.211253    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf2gc\" (UniqueName: \"kubernetes.io/projected/6adcf1d7-002a-4c06-9f78-0b5af2347bcf-kube-api-access-sf2gc\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.639293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.639257    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"
Oct 29 22:49:22.639764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.639283    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27" event={"ID":"49c089e2-11bb-486d-a9e2-1eb04faa066f","Type":"ContainerDied","Data":"656f5dc147b7c1409308bbfe38a2d778da8881ea4d6a6694bbf216af2f1d4f89"}
Oct 29 22:49:22.639764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.639383    4451 scope.go:122] "RemoveContainer" containerID="0c0390d9668f1b1ea482e5c08cd26876675d2638e652e7669ec6013c4a7e0a3a"
Oct 29 22:49:22.640959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.640899    4451 generic.go:356] "Generic (PLEG): container finished" podID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerID="d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27" exitCode=0
Oct 29 22:49:22.641031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.640984    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" event={"ID":"6adcf1d7-002a-4c06-9f78-0b5af2347bcf","Type":"ContainerDied","Data":"d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27"}
Oct 29 22:49:22.641073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.641047    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s" event={"ID":"6adcf1d7-002a-4c06-9f78-0b5af2347bcf","Type":"ContainerDied","Data":"12850dc823ce7a024cfda907a3d26ee5ad3098e230170829e182f70fe7b276e3"}
Oct 29 22:49:22.641073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.640993    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"
Oct 29 22:49:22.649967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.649946    4451 scope.go:122] "RemoveContainer" containerID="d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27"
Oct 29 22:49:22.659043 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.659023    4451 scope.go:122] "RemoveContainer" containerID="d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27"
Oct 29 22:49:22.659295 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:22.659274    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27\": container with ID starting with d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27 not found: ID does not exist" containerID="d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27"
Oct 29 22:49:22.659358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.659307    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27"} err="failed to get container status \"d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27\": rpc error: code = NotFound desc = could not find container \"d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27\": container with ID starting with d3d3a33b72b9409360b6cda420af7f2b5e79b73b38645c35db566ef7bd064e27 not found: ID does not exist"
Oct 29 22:49:22.698281 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.698257    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"]
Oct 29 22:49:22.706097 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.706076    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-nbz7s"]
Oct 29 22:49:22.735285 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.735261    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"]
Oct 29 22:49:22.739738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:22.739720    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-cjl27"]
Oct 29 22:49:24.164176 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:24.164135    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" path="/var/lib/kubelet/pods/49c089e2-11bb-486d-a9e2-1eb04faa066f/volumes"
Oct 29 22:49:24.164869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:24.164849    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" path="/var/lib/kubelet/pods/6adcf1d7-002a-4c06-9f78-0b5af2347bcf/volumes"
Oct 29 22:49:26.930761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:26.930723    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"]
Oct 29 22:49:26.931224 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:26.930975    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" podUID="e3e8c40a-295b-4149-8b4d-25086ce59925" containerName="maas-api" containerID="cri-o://4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b" gracePeriod=30
Oct 29 22:49:27.299647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.299612    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:49:27.496986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.495890    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c\" (UniqueName: \"kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c\") pod \"e3e8c40a-295b-4149-8b4d-25086ce59925\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") "
Oct 29 22:49:27.496986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.496030    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\") pod \"e3e8c40a-295b-4149-8b4d-25086ce59925\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") "
Oct 29 22:49:27.496986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.496088    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs\") pod \"e3e8c40a-295b-4149-8b4d-25086ce59925\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") "
Oct 29 22:49:27.496986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.496137    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca\") pod \"e3e8c40a-295b-4149-8b4d-25086ce59925\" (UID: \"e3e8c40a-295b-4149-8b4d-25086ce59925\") "
Oct 29 22:49:27.499412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.497806    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca" pod "e3e8c40a-295b-4149-8b4d-25086ce59925" (UID: "e3e8c40a-295b-4149-8b4d-25086ce59925"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:27.516157 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.516117    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c" pod "e3e8c40a-295b-4149-8b4d-25086ce59925" (UID: "e3e8c40a-295b-4149-8b4d-25086ce59925"). InnerVolumeSpecName "kube-api-access-28r9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:27.516318 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.516213    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs" pod "e3e8c40a-295b-4149-8b4d-25086ce59925" (UID: "e3e8c40a-295b-4149-8b4d-25086ce59925"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:27.520632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.520578    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls" pod "e3e8c40a-295b-4149-8b4d-25086ce59925" (UID: "e3e8c40a-295b-4149-8b4d-25086ce59925"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:27.599687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.599645    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:27.599687 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.599686    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e8c40a-295b-4149-8b4d-25086ce59925-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:27.599960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.599702    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28r9c\" (UniqueName: \"kubernetes.io/projected/e3e8c40a-295b-4149-8b4d-25086ce59925-kube-api-access-28r9c\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:27.599960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.599715    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e8c40a-295b-4149-8b4d-25086ce59925-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:27.680724 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.680678    4451 generic.go:356] "Generic (PLEG): container finished" podID="e3e8c40a-295b-4149-8b4d-25086ce59925" containerID="4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b" exitCode=0
Oct 29 22:49:27.680972 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.680878    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" event={"ID":"e3e8c40a-295b-4149-8b4d-25086ce59925","Type":"ContainerDied","Data":"4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b"}
Oct 29 22:49:27.680972 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.680944    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j" event={"ID":"e3e8c40a-295b-4149-8b4d-25086ce59925","Type":"ContainerDied","Data":"6c42170b6e771c8a9102e164c662d4110474195457c9dcf58d5d7a214afb75b7"}
Oct 29 22:49:27.680972 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.680970    4451 scope.go:122] "RemoveContainer" containerID="4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b"
Oct 29 22:49:27.681192 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.681057    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"
Oct 29 22:49:27.694084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.694059    4451 scope.go:122] "RemoveContainer" containerID="4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b"
Oct 29 22:49:27.694473 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:27.694407    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b\": container with ID starting with 4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b not found: ID does not exist" containerID="4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b"
Oct 29 22:49:27.694473 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.694448    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b"} err="failed to get container status \"4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b\": rpc error: code = NotFound desc = could not find container \"4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b\": container with ID starting with 4440c834768f585386efe37f8ef2346d698936d8efe58c0387a8a3a8f7dbea3b not found: ID does not exist"
Oct 29 22:49:27.754831 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.754735    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"]
Oct 29 22:49:27.762518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:27.762479    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a69c9-77b4c66946-fj28j"]
Oct 29 22:49:28.165300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:28.165264    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e8c40a-295b-4149-8b4d-25086ce59925" path="/var/lib/kubelet/pods/e3e8c40a-295b-4149-8b4d-25086ce59925/volumes"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.379432    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"]
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380356    4451 cpu_manager.go:414] "containerMap: removing container" podUID="e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" containerName="main"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380377    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" containerName="main"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380405    4451 cpu_manager.go:414] "containerMap: removing container" podUID="e3e8c40a-295b-4149-8b4d-25086ce59925" containerName="maas-api"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380414    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e8c40a-295b-4149-8b4d-25086ce59925" containerName="maas-api"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380459    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380468    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380481    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b2c68168-f92c-4425-a8d5-843d540ebb90" containerName="main"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380489    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c68168-f92c-4425-a8d5-843d540ebb90" containerName="main"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380511    4451 cpu_manager.go:414] "containerMap: removing container" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380518    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380652    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ac0379-0e59-4ea2-a8b0-3a1cd844441b" containerName="main"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380664    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e8c40a-295b-4149-8b4d-25086ce59925" containerName="maas-api"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380676    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adcf1d7-002a-4c06-9f78-0b5af2347bcf" containerName="istio-proxy"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380682    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c68168-f92c-4425-a8d5-843d540ebb90" containerName="main"
Oct 29 22:49:29.383974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.380689    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c089e2-11bb-486d-a9e2-1eb04faa066f" containerName="istio-proxy"
Oct 29 22:49:29.385882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.385858    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.403112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.401426    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"]
Oct 29 22:49:29.406370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.406345    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"openshift-service-ca.crt\""
Oct 29 22:49:29.407103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.407076    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"kube-root-ca.crt\""
Oct 29 22:49:29.412021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.411981    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"default-dockercfg-p84j2\""
Oct 29 22:49:29.412021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.411989    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"auth-test-model-af260f-kserve-self-signed-certs\""
Oct 29 22:49:29.437163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.437130    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.437316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.437300    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.437385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.437370    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.437456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.437429    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fz6s\" (UniqueName: \"kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.437546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.437531    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.437619 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.437605    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.538955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.538849    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.539519 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.539465    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.539791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.539757    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.539904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.539797    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fz6s\" (UniqueName: \"kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.539904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.539845    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.540050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.539940    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.540050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.540014    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.540401 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.540376    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.540575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.540546    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.541752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.541730    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.542839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.542810    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.557446 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.557424    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fz6s\" (UniqueName: \"kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s\") pod \"auth-test-model-af260f-kserve-d854864c7-cd9sp\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") " pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:29.763571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:29.728168    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:30.140589 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:30.140556    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"]
Oct 29 22:49:30.141450 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:49:30.141424    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0254ffd1_ce60_471d_811e_3770cc3082b8.slice/crio-b1178ddb9e3debf148d00f3357db4644c5d3af29ba65c972643d9d7d137ac275 WatchSource:0}: Error finding container b1178ddb9e3debf148d00f3357db4644c5d3af29ba65c972643d9d7d137ac275: Status 404 returned error can't find the container with id b1178ddb9e3debf148d00f3357db4644c5d3af29ba65c972643d9d7d137ac275
Oct 29 22:49:30.816538 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:30.816499    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" event={"ID":"0254ffd1-ce60-471d-811e-3770cc3082b8","Type":"ContainerStarted","Data":"d5e927dc19c7185a001fb1002dcc568e726dd848a8180692735a07aac64d0693"}
Oct 29 22:49:30.817012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:30.816561    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" event={"ID":"0254ffd1-ce60-471d-811e-3770cc3082b8","Type":"ContainerStarted","Data":"b1178ddb9e3debf148d00f3357db4644c5d3af29ba65c972643d9d7d137ac275"}
Oct 29 22:49:30.855718 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:30.855668    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" podStartSLOduration=1.855653494 podStartE2EDuration="1.855653494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:30.852759141 +0000 UTC m=+2110.795717848" watchObservedRunningTime="2026-10-29 22:49:30.855653494 +0000 UTC m=+2110.798612198"
Oct 29 22:49:31.729009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:31.728972    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:39.729359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:39.729282    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:39.742158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:39.742134    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:39.917338 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:39.917309    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:49:40.265068 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.265035    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"]
Oct 29 22:49:40.265301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.265253    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" podUID="f80a04e9-d827-4c3e-b23c-ec35d86078d9" containerName="maas-api" containerID="cri-o://7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10" gracePeriod=30
Oct 29 22:49:40.705109 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.701996    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:49:40.780986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.770382    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k\" (UniqueName: \"kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k\") pod \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") "
Oct 29 22:49:40.781650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.781623    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\") pod \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") "
Oct 29 22:49:40.790221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.786052    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs\") pod \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") "
Oct 29 22:49:40.790221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.786334    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca\") pod \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\" (UID: \"f80a04e9-d827-4c3e-b23c-ec35d86078d9\") "
Oct 29 22:49:40.790221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.790161    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k" pod "f80a04e9-d827-4c3e-b23c-ec35d86078d9" (UID: "f80a04e9-d827-4c3e-b23c-ec35d86078d9"). InnerVolumeSpecName "kube-api-access-dk92k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:40.821026 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.820959    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs" pod "f80a04e9-d827-4c3e-b23c-ec35d86078d9" (UID: "f80a04e9-d827-4c3e-b23c-ec35d86078d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:40.842367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.842332    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls" pod "f80a04e9-d827-4c3e-b23c-ec35d86078d9" (UID: "f80a04e9-d827-4c3e-b23c-ec35d86078d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:40.850771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.850735    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca" pod "f80a04e9-d827-4c3e-b23c-ec35d86078d9" (UID: "f80a04e9-d827-4c3e-b23c-ec35d86078d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:40.888400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.888365    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk92k\" (UniqueName: \"kubernetes.io/projected/f80a04e9-d827-4c3e-b23c-ec35d86078d9-kube-api-access-dk92k\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.888400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.888396    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.888400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.888409    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80a04e9-d827-4c3e-b23c-ec35d86078d9-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.888641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.888418    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80a04e9-d827-4c3e-b23c-ec35d86078d9-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.914238 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.914149    4451 generic.go:356] "Generic (PLEG): container finished" podID="f80a04e9-d827-4c3e-b23c-ec35d86078d9" containerID="7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10" exitCode=0
Oct 29 22:49:40.914393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.914252    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"
Oct 29 22:49:40.914393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.914256    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" event={"ID":"f80a04e9-d827-4c3e-b23c-ec35d86078d9","Type":"ContainerDied","Data":"7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10"}
Oct 29 22:49:40.914393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.914327    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh" event={"ID":"f80a04e9-d827-4c3e-b23c-ec35d86078d9","Type":"ContainerDied","Data":"40e21d9f696ae65c783b932b370c1d0ba7a883d1a01287892f0c73c66b1d37ee"}
Oct 29 22:49:40.914393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.914359    4451 scope.go:122] "RemoveContainer" containerID="7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10"
Oct 29 22:49:40.928700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.928664    4451 scope.go:122] "RemoveContainer" containerID="7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10"
Oct 29 22:49:40.929278 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:40.929246    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10\": container with ID starting with 7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10 not found: ID does not exist" containerID="7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10"
Oct 29 22:49:40.929383 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.929279    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10"} err="failed to get container status \"7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10\": rpc error: code = NotFound desc = could not find container \"7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10\": container with ID starting with 7bf0bf1b18c51e482db34991a00e1a4b2b1ab0f7550461a5efae5ae8df9b2b10 not found: ID does not exist"
Oct 29 22:49:40.964537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.964503    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"]
Oct 29 22:49:40.967471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:40.967449    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8147a8-745679f4fc-gxwvh"]
Oct 29 22:49:42.164576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:42.164542    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80a04e9-d827-4c3e-b23c-ec35d86078d9" path="/var/lib/kubelet/pods/f80a04e9-d827-4c3e-b23c-ec35d86078d9/volumes"
Oct 29 22:49:45.028817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.028778    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"]
Oct 29 22:49:45.029592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.029569    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f80a04e9-d827-4c3e-b23c-ec35d86078d9" containerName="maas-api"
Oct 29 22:49:45.029592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.029591    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80a04e9-d827-4c3e-b23c-ec35d86078d9" containerName="maas-api"
Oct 29 22:49:45.029796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.029781    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80a04e9-d827-4c3e-b23c-ec35d86078d9" containerName="maas-api"
Oct 29 22:49:45.034490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.034459    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.055017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.054989    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"]
Oct 29 22:49:45.060690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.060636    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"kube-root-ca.crt\""
Oct 29 22:49:45.060690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.060636    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"openshift-service-ca.crt\""
Oct 29 22:49:45.060690 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.060636    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"default-dockercfg-75x84\""
Oct 29 22:49:45.060906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.060758    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-9aafe6\"/\"rate-test-model-9aafe6-kserve-self-signed-certs\""
Oct 29 22:49:45.148721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.148691    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.148895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.148863    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.148998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.148899    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.149040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.149014    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.149086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.149048    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjsz\" (UniqueName: \"kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.149248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.149224    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250371    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250608 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250431    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250626    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250687    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250773    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250814    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.250971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.250820    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjsz\" (UniqueName: \"kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.251670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.251640    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.251750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.251672    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.253545 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.253520    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.253742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.253719    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.270931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.270264    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjsz\" (UniqueName: \"kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz\") pod \"rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") " pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.356348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.356312    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:45.680023 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:49:45.679989    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f67cc43_b89b_4b47_9e9a_9e506dca8a23.slice/crio-63fee78ef3f99971d566f22e14775901610c2d760e9c0ca40e5598c3069429f3 WatchSource:0}: Error finding container 63fee78ef3f99971d566f22e14775901610c2d760e9c0ca40e5598c3069429f3: Status 404 returned error can't find the container with id 63fee78ef3f99971d566f22e14775901610c2d760e9c0ca40e5598c3069429f3
Oct 29 22:49:45.680324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.680301    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"]
Oct 29 22:49:45.733025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.732994    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"]
Oct 29 22:49:45.733491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.733219    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" podUID="ebab7eb5-9446-43fd-9210-9f33a470640a" containerName="maas-api" containerID="cri-o://970cd5ec8b1708398974e4a15a4b4e9fbf7cc3565c725df6703ab1fc4bfbf9e0" gracePeriod=30
Oct 29 22:49:45.953801 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.953719    4451 generic.go:356] "Generic (PLEG): container finished" podID="ebab7eb5-9446-43fd-9210-9f33a470640a" containerID="970cd5ec8b1708398974e4a15a4b4e9fbf7cc3565c725df6703ab1fc4bfbf9e0" exitCode=0
Oct 29 22:49:45.953980 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.953818    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" event={"ID":"ebab7eb5-9446-43fd-9210-9f33a470640a","Type":"ContainerDied","Data":"970cd5ec8b1708398974e4a15a4b4e9fbf7cc3565c725df6703ab1fc4bfbf9e0"}
Oct 29 22:49:45.955827 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.955793    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" event={"ID":"4f67cc43-b89b-4b47-9e9a-9e506dca8a23","Type":"ContainerStarted","Data":"340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd"}
Oct 29 22:49:45.955968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:45.955839    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" event={"ID":"4f67cc43-b89b-4b47-9e9a-9e506dca8a23","Type":"ContainerStarted","Data":"63fee78ef3f99971d566f22e14775901610c2d760e9c0ca40e5598c3069429f3"}
Oct 29 22:49:46.000817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.000733    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" podStartSLOduration=1.000713552 podStartE2EDuration="1.000713552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:45.998878709 +0000 UTC m=+2125.941837414" watchObservedRunningTime="2026-10-29 22:49:46.000713552 +0000 UTC m=+2125.943672258"
Oct 29 22:49:46.039307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.039275    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:49:46.170666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.170634    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8\" (UniqueName: \"kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8\") pod \"ebab7eb5-9446-43fd-9210-9f33a470640a\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") "
Oct 29 22:49:46.170961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.170895    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs\") pod \"ebab7eb5-9446-43fd-9210-9f33a470640a\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") "
Oct 29 22:49:46.171285 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.171263    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca\") pod \"ebab7eb5-9446-43fd-9210-9f33a470640a\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") "
Oct 29 22:49:46.171380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.171362    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\") pod \"ebab7eb5-9446-43fd-9210-9f33a470640a\" (UID: \"ebab7eb5-9446-43fd-9210-9f33a470640a\") "
Oct 29 22:49:46.171698 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.171671    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca" pod "ebab7eb5-9446-43fd-9210-9f33a470640a" (UID: "ebab7eb5-9446-43fd-9210-9f33a470640a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:46.172504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.172480    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab7eb5-9446-43fd-9210-9f33a470640a-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:46.174984 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.174744    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8" pod "ebab7eb5-9446-43fd-9210-9f33a470640a" (UID: "ebab7eb5-9446-43fd-9210-9f33a470640a"). InnerVolumeSpecName "kube-api-access-9c2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:46.175116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.175045    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs" pod "ebab7eb5-9446-43fd-9210-9f33a470640a" (UID: "ebab7eb5-9446-43fd-9210-9f33a470640a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:46.175187 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.175137    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls" pod "ebab7eb5-9446-43fd-9210-9f33a470640a" (UID: "ebab7eb5-9446-43fd-9210-9f33a470640a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:46.276457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.275300    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2l8\" (UniqueName: \"kubernetes.io/projected/ebab7eb5-9446-43fd-9210-9f33a470640a-kube-api-access-9c2l8\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:46.276457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.275338    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:46.276457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.275352    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebab7eb5-9446-43fd-9210-9f33a470640a-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:46.357020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.356990    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:46.963427 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.963391    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"
Oct 29 22:49:46.963600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.963426    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f" event={"ID":"ebab7eb5-9446-43fd-9210-9f33a470640a","Type":"ContainerDied","Data":"d6676e502e81a0813c79ed45593d9beb72739cac5ebaef11feaa361c3b90164e"}
Oct 29 22:49:46.963600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:46.963489    4451 scope.go:122] "RemoveContainer" containerID="970cd5ec8b1708398974e4a15a4b4e9fbf7cc3565c725df6703ab1fc4bfbf9e0"
Oct 29 22:49:47.014599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:47.014567    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"]
Oct 29 22:49:47.024634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:47.024608    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e619b6-78864d877f-t9f9f"]
Oct 29 22:49:48.165669 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:48.165632    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebab7eb5-9446-43fd-9210-9f33a470640a" path="/var/lib/kubelet/pods/ebab7eb5-9446-43fd-9210-9f33a470640a/volumes"
Oct 29 22:49:48.743200 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:48.743156    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:49:50.651276 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.651241    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"]
Oct 29 22:49:50.652617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.652591    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ebab7eb5-9446-43fd-9210-9f33a470640a" containerName="maas-api"
Oct 29 22:49:50.652617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.652616    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebab7eb5-9446-43fd-9210-9f33a470640a" containerName="maas-api"
Oct 29 22:49:50.652766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.652707    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebab7eb5-9446-43fd-9210-9f33a470640a" containerName="maas-api"
Oct 29 22:49:50.657668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.657648    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.680755 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.680717    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-6fc6bd-openshift-default-dockercfg-mff5d\""
Oct 29 22:49:50.687559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.687525    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"]
Oct 29 22:49:50.747395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747360    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747413    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747517    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747659 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747590    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747659 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747645    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrfr\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747782 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747765    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747820 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747794    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747818    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.747921 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.747879    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.848994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.848963    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849014    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849056    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849139    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849162    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrfr\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849321 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849277    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849330    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849433    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849443    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849477    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849562    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849678    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849860    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.849973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.849899    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.851503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.851479    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.851776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.851759    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.866202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.866177    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.866363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.866273    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrfr\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:50.972638 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:50.972545    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:51.290564 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.290525    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"]
Oct 29 22:49:51.290742 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:49:51.290720    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff55386_965c_4a74_9e62_fd0b912ef936.slice/crio-95046d33d7a9a18a2bf3880324cde152c38466cb765a9cd0eca34817e5220e46 WatchSource:0}: Error finding container 95046d33d7a9a18a2bf3880324cde152c38466cb765a9cd0eca34817e5220e46: Status 404 returned error can't find the container with id 95046d33d7a9a18a2bf3880324cde152c38466cb765a9cd0eca34817e5220e46
Oct 29 22:49:51.294209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.294176    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:49:51.294311 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.294256    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:49:51.294377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.294309    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:49:51.917950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.916962    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"]
Oct 29 22:49:51.928533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.928472    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:51.939656 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.939623    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"]
Oct 29 22:49:51.944947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.944344    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa3c6\"/\"kube-root-ca.crt\""
Oct 29 22:49:51.944947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.944375    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa3c6\"/\"default-dockercfg-9l4x2\""
Oct 29 22:49:51.944947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.944411    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa3c6\"/\"openshift-service-ca.crt\""
Oct 29 22:49:51.944947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.944582    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa3c6\"/\"sub-test-model-aaa3c6-kserve-self-signed-certs\""
Oct 29 22:49:51.968456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.968410    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:51.968594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.968541    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:51.968652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.968607    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhr99\" (UniqueName: \"kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:51.968729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.968710    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:51.968786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.968760    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:51.968838 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:51.968816    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.038508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.038463    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" event={"ID":"5ff55386-965c-4a74-9e62-fd0b912ef936","Type":"ContainerStarted","Data":"f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8"}
Oct 29 22:49:52.038675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.038573    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" event={"ID":"5ff55386-965c-4a74-9e62-fd0b912ef936","Type":"ContainerStarted","Data":"95046d33d7a9a18a2bf3880324cde152c38466cb765a9cd0eca34817e5220e46"}
Oct 29 22:49:52.070529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070492    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.070529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070532    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.070785 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070559    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.070785 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070688    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.070785 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070733    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.070785 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070765    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhr99\" (UniqueName: \"kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.071017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.070998    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.071462 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.071247    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.071462 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.071419    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.073330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.073302    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.074180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.074057    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.118941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.114422    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhr99\" (UniqueName: \"kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99\") pod \"sub-test-model-aaa3c6-kserve-97758679f-tvm97\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") " pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.261259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.261164    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:52.660133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.660084    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podStartSLOduration=2.660067824 podStartE2EDuration="2.660067824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:52.083898913 +0000 UTC m=+2132.026857620" watchObservedRunningTime="2026-10-29 22:49:52.660067824 +0000 UTC m=+2132.603026528"
Oct 29 22:49:52.661516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.661489    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"]
Oct 29 22:49:52.661857 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:49:52.661834    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod150ac800_4c96_4bb4_af90_c08ca6c25fbf.slice/crio-b64c535a332fdd4e59c7c3d629b7b986fd512294cb6ebbfff66a3b48853d92aa WatchSource:0}: Error finding container b64c535a332fdd4e59c7c3d629b7b986fd512294cb6ebbfff66a3b48853d92aa: Status 404 returned error can't find the container with id b64c535a332fdd4e59c7c3d629b7b986fd512294cb6ebbfff66a3b48853d92aa
Oct 29 22:49:52.973277 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.973195    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:52.974677 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.974655    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:49:52.974813 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:52.974707    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:49:53.050775 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.050393    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97" event={"ID":"150ac800-4c96-4bb4-af90-c08ca6c25fbf","Type":"ContainerStarted","Data":"884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1"}
Oct 29 22:49:53.050775 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.050441    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97" event={"ID":"150ac800-4c96-4bb4-af90-c08ca6c25fbf","Type":"ContainerStarted","Data":"b64c535a332fdd4e59c7c3d629b7b986fd512294cb6ebbfff66a3b48853d92aa"}
Oct 29 22:49:53.052460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.052430    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"]
Oct 29 22:49:53.052614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.052595    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" podUID="f6207ed6-53f4-47cd-b253-907cdb512979" containerName="maas-api" containerID="cri-o://1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd" gracePeriod=30
Oct 29 22:49:53.126935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.100434    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97" podStartSLOduration=2.100415764 podStartE2EDuration="2.100415764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:53.098318603 +0000 UTC m=+2133.041277309" watchObservedRunningTime="2026-10-29 22:49:53.100415764 +0000 UTC m=+2133.043374470"
Oct 29 22:49:53.262610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.262527    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:49:53.850213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.850174    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:49:53.856968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.856933    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls\") pod \"f6207ed6-53f4-47cd-b253-907cdb512979\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") "
Oct 29 22:49:53.857130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.856988    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5\" (UniqueName: \"kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5\") pod \"f6207ed6-53f4-47cd-b253-907cdb512979\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") "
Oct 29 22:49:53.857130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.857011    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs\") pod \"f6207ed6-53f4-47cd-b253-907cdb512979\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") "
Oct 29 22:49:53.857130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.857035    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca\") pod \"f6207ed6-53f4-47cd-b253-907cdb512979\" (UID: \"f6207ed6-53f4-47cd-b253-907cdb512979\") "
Oct 29 22:49:53.858187 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.858162    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca" pod "f6207ed6-53f4-47cd-b253-907cdb512979" (UID: "f6207ed6-53f4-47cd-b253-907cdb512979"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:53.860949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.860898    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs" pod "f6207ed6-53f4-47cd-b253-907cdb512979" (UID: "f6207ed6-53f4-47cd-b253-907cdb512979"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:53.861090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.861062    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5" pod "f6207ed6-53f4-47cd-b253-907cdb512979" (UID: "f6207ed6-53f4-47cd-b253-907cdb512979"). InnerVolumeSpecName "kube-api-access-zlhs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:53.861202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.861185    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls" pod "f6207ed6-53f4-47cd-b253-907cdb512979" (UID: "f6207ed6-53f4-47cd-b253-907cdb512979"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:53.959356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.959307    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.959356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.959356    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhs5\" (UniqueName: \"kubernetes.io/projected/f6207ed6-53f4-47cd-b253-907cdb512979-kube-api-access-zlhs5\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.959644 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.959370    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6207ed6-53f4-47cd-b253-907cdb512979-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.959644 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.959384    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6207ed6-53f4-47cd-b253-907cdb512979-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.980605 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.975384    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:49:53.980605 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:53.975446    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:49:54.027996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.026550    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"]
Oct 29 22:49:54.027996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.026808    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy" containerID="cri-o://552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1" gracePeriod=30
Oct 29 22:49:54.057458 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.057427    4451 generic.go:356] "Generic (PLEG): container finished" podID="f6207ed6-53f4-47cd-b253-907cdb512979" containerID="1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd" exitCode=0
Oct 29 22:49:54.058223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.058200    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"
Oct 29 22:49:54.060111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.060083    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" event={"ID":"f6207ed6-53f4-47cd-b253-907cdb512979","Type":"ContainerDied","Data":"1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd"}
Oct 29 22:49:54.060214 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.060135    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" event={"ID":"f6207ed6-53f4-47cd-b253-907cdb512979","Type":"ContainerDied","Data":"cf39a0dafdd0260d1ed4d57d0ecf9e52a887f016d025147b1285b5fcb53154a9"}
Oct 29 22:49:54.060214 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.060157    4451 scope.go:122] "RemoveContainer" containerID="1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd"
Oct 29 22:49:54.079302 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.079266    4451 scope.go:122] "RemoveContainer" containerID="1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd"
Oct 29 22:49:54.079645 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:49:54.079624    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd\": container with ID starting with 1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd not found: ID does not exist" containerID="1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd"
Oct 29 22:49:54.079718 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.079653    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd"} err="failed to get container status \"1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd\": rpc error: code = NotFound desc = could not find container \"1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd\": container with ID starting with 1199b653e63e667e4b1db878cd38ba9366378592272d065b1f23d8a41b4b00dd not found: ID does not exist"
Oct 29 22:49:54.152001 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.151031    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"]
Oct 29 22:49:54.166943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.154278    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7"]
Oct 29 22:49:54.232607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.232570    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6207ed6-53f4-47cd-b253-907cdb512979" path="/var/lib/kubelet/pods/f6207ed6-53f4-47cd-b253-907cdb512979/volumes"
Oct 29 22:49:54.973640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.973607    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:49:54.973818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:54.973661    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:49:55.357230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:55.357192    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:55.374700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:55.374672    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:55.973600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:55.973567    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:49:55.973760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:55.973630    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:49:56.088462 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:56.088433    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:49:56.435938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:56.434006    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a3ef0-865cd4d677-9mww7" podUID="f6207ed6-53f4-47cd-b253-907cdb512979" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.97:8443/health\": context deadline exceeded"
Oct 29 22:49:56.973460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:56.973423    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:49:56.973643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:56.973483    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:49:58.066159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:58.066126    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:58.066657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:58.066513    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:58.067613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:58.067587    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 22:49:59.320266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.320244    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:49:59.397883 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.397844    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.397974    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398034    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398070    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398269 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398165    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398329 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398283    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398355    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398454    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.398748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398730    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy\") pod \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\" (UID: \"b14413f3-c6ae-4f73-8bcd-bedc843cfba7\") "
Oct 29 22:49:59.399457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.398818    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.399604 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.399090    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.399731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.399343    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.399731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.399403    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.399995 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.399967    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:59.400145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.400039    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.400291 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.400269    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.400411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.400397    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.400522 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.400509    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.400788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.400701    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.400788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.400748    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "kube-api-access-qxnct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.401395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.401364    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.402193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.402165    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo" pod "b14413f3-c6ae-4f73-8bcd-bedc843cfba7" (UID: "b14413f3-c6ae-4f73-8bcd-bedc843cfba7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:59.502208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.502171    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnct\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-kube-api-access-qxnct\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.502208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.502202    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.502208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.502216    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.502444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.502228    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.502444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:49:59.502239    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14413f3-c6ae-4f73-8bcd-bedc843cfba7-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:00.100522 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.100485    4451 generic.go:356] "Generic (PLEG): container finished" podID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerID="552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1" exitCode=0
Oct 29 22:50:00.100701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.100555    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"
Oct 29 22:50:00.100701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.100575    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" event={"ID":"b14413f3-c6ae-4f73-8bcd-bedc843cfba7","Type":"ContainerDied","Data":"552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1"}
Oct 29 22:50:00.100701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.100663    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7" event={"ID":"b14413f3-c6ae-4f73-8bcd-bedc843cfba7","Type":"ContainerDied","Data":"5c38a900ea61d00d5bc591bc22a9886ebde1bf3ea134453ce92c5639b48a7b2f"}
Oct 29 22:50:00.100701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.100695    4451 scope.go:122] "RemoveContainer" containerID="552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1"
Oct 29 22:50:00.111802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.111785    4451 scope.go:122] "RemoveContainer" containerID="552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1"
Oct 29 22:50:00.112132 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:00.112111    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1\": container with ID starting with 552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1 not found: ID does not exist" containerID="552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1"
Oct 29 22:50:00.112186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.112142    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1"} err="failed to get container status \"552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1\": rpc error: code = NotFound desc = could not find container \"552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1\": container with ID starting with 552862a45f5e240f7792e1539d734d473957a1bcf8619571fe82aae737b5dcb1 not found: ID does not exist"
Oct 29 22:50:00.169484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.169457    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"]
Oct 29 22:50:00.172613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:00.172590    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-59rv7"]
Oct 29 22:50:02.164717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:02.164685    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" path="/var/lib/kubelet/pods/b14413f3-c6ae-4f73-8bcd-bedc843cfba7/volumes"
Oct 29 22:50:02.262109 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:02.262078    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:50:02.274880 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:02.274856    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:50:03.137841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:03.137804    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:50:06.315475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.315435    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"]
Oct 29 22:50:06.315979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.315679    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g" podUID="555d79bc-ed00-47ea-a390-0744438d6b82" containerName="maas-api" containerID="cri-o://ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec" gracePeriod=30
Oct 29 22:50:06.786142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.786096    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:50:06.851029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.850714    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs\") pod \"555d79bc-ed00-47ea-a390-0744438d6b82\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") "
Oct 29 22:50:06.851029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.850943    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf\" (UniqueName: \"kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf\") pod \"555d79bc-ed00-47ea-a390-0744438d6b82\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") "
Oct 29 22:50:06.851029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.851016    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls\") pod \"555d79bc-ed00-47ea-a390-0744438d6b82\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") "
Oct 29 22:50:06.851343 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.851053    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca\") pod \"555d79bc-ed00-47ea-a390-0744438d6b82\" (UID: \"555d79bc-ed00-47ea-a390-0744438d6b82\") "
Oct 29 22:50:06.854940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.852681    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca" pod "555d79bc-ed00-47ea-a390-0744438d6b82" (UID: "555d79bc-ed00-47ea-a390-0744438d6b82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:06.855937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.855340    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf" pod "555d79bc-ed00-47ea-a390-0744438d6b82" (UID: "555d79bc-ed00-47ea-a390-0744438d6b82"). InnerVolumeSpecName "kube-api-access-ckpqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:06.867200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.865110    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs" pod "555d79bc-ed00-47ea-a390-0744438d6b82" (UID: "555d79bc-ed00-47ea-a390-0744438d6b82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:50:06.867200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.866117    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls" pod "555d79bc-ed00-47ea-a390-0744438d6b82" (UID: "555d79bc-ed00-47ea-a390-0744438d6b82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:50:06.954805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.954717    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555d79bc-ed00-47ea-a390-0744438d6b82-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:06.954805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.954761    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:06.954805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.954776    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckpqf\" (UniqueName: \"kubernetes.io/projected/555d79bc-ed00-47ea-a390-0744438d6b82-kube-api-access-ckpqf\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:06.954805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:06.954790    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555d79bc-ed00-47ea-a390-0744438d6b82-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.173299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.170225    4451 generic.go:356] "Generic (PLEG): container finished" podID="555d79bc-ed00-47ea-a390-0744438d6b82" containerID="ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec" exitCode=0
Oct 29 22:50:07.173299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.170299    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g" event={"ID":"555d79bc-ed00-47ea-a390-0744438d6b82","Type":"ContainerDied","Data":"ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec"}
Oct 29 22:50:07.173299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.170352    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g" event={"ID":"555d79bc-ed00-47ea-a390-0744438d6b82","Type":"ContainerDied","Data":"eb6bdf442e1937f6605f1c43f0953df5fb823695acc72c91507ae980c7729751"}
Oct 29 22:50:07.173299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.170383    4451 scope.go:122] "RemoveContainer" containerID="ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec"
Oct 29 22:50:07.173299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.170386    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"
Oct 29 22:50:07.191957 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.189062    4451 scope.go:122] "RemoveContainer" containerID="ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec"
Oct 29 22:50:07.191957 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:07.189402    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec\": container with ID starting with ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec not found: ID does not exist" containerID="ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec"
Oct 29 22:50:07.191957 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.189434    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec"} err="failed to get container status \"ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec\": rpc error: code = NotFound desc = could not find container \"ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec\": container with ID starting with ddf98aac2c2f478f8353d8fb5cc9e1c3bef7e8eaa56333322a8fc3c0a36556ec not found: ID does not exist"
Oct 29 22:50:07.222974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.222503    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"]
Oct 29 22:50:07.225833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:07.225806    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05417d-559cbc98d5-vrv8g"]
Oct 29 22:50:08.164906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:08.164874    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555d79bc-ed00-47ea-a390-0744438d6b82" path="/var/lib/kubelet/pods/555d79bc-ed00-47ea-a390-0744438d6b82/volumes"
Oct 29 22:50:18.645400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.645368    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"]
Oct 29 22:50:18.646000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.645984    4451 cpu_manager.go:414] "containerMap: removing container" podUID="555d79bc-ed00-47ea-a390-0744438d6b82" containerName="maas-api"
Oct 29 22:50:18.646000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646000    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555d79bc-ed00-47ea-a390-0744438d6b82" containerName="maas-api"
Oct 29 22:50:18.646115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646021    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy"
Oct 29 22:50:18.646115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646026    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy"
Oct 29 22:50:18.646115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646036    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f6207ed6-53f4-47cd-b253-907cdb512979" containerName="maas-api"
Oct 29 22:50:18.646115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646042    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6207ed6-53f4-47cd-b253-907cdb512979" containerName="maas-api"
Oct 29 22:50:18.646233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646132    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14413f3-c6ae-4f73-8bcd-bedc843cfba7" containerName="istio-proxy"
Oct 29 22:50:18.646233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646143    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6207ed6-53f4-47cd-b253-907cdb512979" containerName="maas-api"
Oct 29 22:50:18.646233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.646161    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="555d79bc-ed00-47ea-a390-0744438d6b82" containerName="maas-api"
Oct 29 22:50:18.649156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.649134    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.667922 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.667872    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"kube-root-ca.crt\""
Oct 29 22:50:18.669430 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.669405    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"]
Oct 29 22:50:18.674907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.674884    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"openshift-service-ca.crt\""
Oct 29 22:50:18.675286 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.674973    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"auth-test-model-d68c2c-kserve-self-signed-certs\""
Oct 29 22:50:18.678165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.678142    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"default-dockercfg-kgqsz\""
Oct 29 22:50:18.747051 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:18.747010    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:50:18.748702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.748678    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.748896 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.748876    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.749084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.749062    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.749158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.749098    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.749158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.749124    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.749269 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.749244    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qjdv\" (UniqueName: \"kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.851275 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.851229    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.851472 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.851449    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.851599 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.851580    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.851670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.851610    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.851670 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.851636    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.851776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.851682    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qjdv\" (UniqueName: \"kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.852177 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.852138    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.852507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.852479    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.852603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.852540    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.853954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.853893    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.854355 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.854333    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.869548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.869521    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qjdv\" (UniqueName: \"kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv\") pod \"auth-test-model-d68c2c-kserve-86575997b5-pz65q\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:18.966013 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:18.964038    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:19.287128 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:19.287100    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"]
Oct 29 22:50:19.287798 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:50:19.287761    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc12bca7d_9928_4dac_89ed_50c31ade29a5.slice/crio-c7092a8398238bcee416a18c12e647f78cd267084eae8895ab540d4a19c341a7 WatchSource:0}: Error finding container c7092a8398238bcee416a18c12e647f78cd267084eae8895ab540d4a19c341a7: Status 404 returned error can't find the container with id c7092a8398238bcee416a18c12e647f78cd267084eae8895ab540d4a19c341a7
Oct 29 22:50:20.254866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:20.254826    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q" event={"ID":"c12bca7d-9928-4dac-89ed-50c31ade29a5","Type":"ContainerStarted","Data":"d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269"}
Oct 29 22:50:20.254866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:20.254871    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q" event={"ID":"c12bca7d-9928-4dac-89ed-50c31ade29a5","Type":"ContainerStarted","Data":"c7092a8398238bcee416a18c12e647f78cd267084eae8895ab540d4a19c341a7"}
Oct 29 22:50:20.292887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:20.292839    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q" podStartSLOduration=2.292825838 podStartE2EDuration="2.292825838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:20.291064516 +0000 UTC m=+2160.234023222" watchObservedRunningTime="2026-10-29 22:50:20.292825838 +0000 UTC m=+2160.235784543"
Oct 29 22:50:20.964725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:20.964696    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:24.553185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.553151    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"]
Oct 29 22:50:24.559438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.559415    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.577081 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.577045    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-06a95f\"/\"kube-root-ca.crt\""
Oct 29 22:50:24.577447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.577397    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-06a95f\"/\"openshift-service-ca.crt\""
Oct 29 22:50:24.581056 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.581026    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"]
Oct 29 22:50:24.644114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.644075    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.644288 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.644125    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7mw\" (UniqueName: \"kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.644417 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.644395    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.644487 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.644438    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.644607 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.644587    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.644652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.644638    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.668121 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.668095    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-06a95f\"/\"auth-test-model-06a95f-kserve-self-signed-certs\""
Oct 29 22:50:24.668121 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.668118    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-06a95f\"/\"default-dockercfg-69k9s\""
Oct 29 22:50:24.745686 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.745650    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.745853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.745712    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.745853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.745753    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.745853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.745791    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7mw\" (UniqueName: \"kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.746058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.745964    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.746058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.745998    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.746174 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.746139    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.746233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.746190    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.746366 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.746342    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.748345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.748317    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.748764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.748741    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.763600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.763575    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7mw\" (UniqueName: \"kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw\") pod \"auth-test-model-06a95f-kserve-5db98dbc57-xpl7w\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") " pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:24.884940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:24.877984    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:25.230266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:25.230176    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"]
Oct 29 22:50:25.234336 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:50:25.234308    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e38a10c_9fbf_478e_9e7c_0951f44eb630.slice/crio-1cddf9a03c318ae85f19f215c51825d0da6f698c6047a1bd0517315f28133adb WatchSource:0}: Error finding container 1cddf9a03c318ae85f19f215c51825d0da6f698c6047a1bd0517315f28133adb: Status 404 returned error can't find the container with id 1cddf9a03c318ae85f19f215c51825d0da6f698c6047a1bd0517315f28133adb
Oct 29 22:50:25.289119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:25.289085    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" event={"ID":"7e38a10c-9fbf-478e-9e7c-0951f44eb630","Type":"ContainerStarted","Data":"1cddf9a03c318ae85f19f215c51825d0da6f698c6047a1bd0517315f28133adb"}
Oct 29 22:50:26.296727 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:26.296687    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" event={"ID":"7e38a10c-9fbf-478e-9e7c-0951f44eb630","Type":"ContainerStarted","Data":"a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d"}
Oct 29 22:50:26.333297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:26.333236    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" podStartSLOduration=2.333218075 podStartE2EDuration="2.333218075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:26.331430227 +0000 UTC m=+2166.274388954" watchObservedRunningTime="2026-10-29 22:50:26.333218075 +0000 UTC m=+2166.276176781"
Oct 29 22:50:26.880110 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:26.880076    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:28.964649 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:28.964612    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:28.981017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:28.980988    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:29.326040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:29.326005    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:50:34.880152 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:34.880114    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:34.892975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:34.892944    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:35.366819 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:35.366791    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:50:39.428583 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:39.428545    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"]
Oct 29 22:50:39.429220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:39.428852    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy" containerID="cri-o://4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4" gracePeriod=30
Oct 29 22:50:39.625411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:39.625376    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"]
Oct 29 22:50:39.625652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:39.625628    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy" containerID="cri-o://fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce" gracePeriod=30
Oct 29 22:50:41.106193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.106154    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"]
Oct 29 22:50:41.111442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.111420    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.136850 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.136812    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-eb02df-serving-cert\""
Oct 29 22:50:41.142123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.142088    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"]
Oct 29 22:50:41.200812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.200782    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.201009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.200971    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.201078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.201039    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4fp\" (UniqueName: \"kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.201180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.201162    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.302998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.302960    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.303186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.303046    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.303186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.303096    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4fp\" (UniqueName: \"kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.303186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.303129    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.303675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.303649    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.305923 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.305876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.306074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.305876    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.320203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.320172    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4fp\" (UniqueName: \"kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp\") pod \"maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.426203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.426114    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:41.633571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.633532    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"]
Oct 29 22:50:41.638862 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.638844    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.658903 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.658875    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"]
Oct 29 22:50:41.709345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709267    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709417    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709466    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvdk\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709582    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709632    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709698    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709770 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709750    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709816 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709772    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.709816 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.709796    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.745119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.745096    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c933cb-openshift-default-dockercfg-f99fc\""
Oct 29 22:50:41.746846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.746825    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"]
Oct 29 22:50:41.747111 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:50:41.747091    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770b4385_960b_4f68_9532_483d0dae622c.slice/crio-cb758a8436753b0e68cae73b813b6dc5bc33a957067d5a53cdcc400920ecf930 WatchSource:0}: Error finding container cb758a8436753b0e68cae73b813b6dc5bc33a957067d5a53cdcc400920ecf930: Status 404 returned error can't find the container with id cb758a8436753b0e68cae73b813b6dc5bc33a957067d5a53cdcc400920ecf930
Oct 29 22:50:41.811227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811194    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811243    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811268    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811299    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811361    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811495    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811561    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvdk\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811602    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811642    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811671    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811710    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.811787 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.811783    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.812256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.812222    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.812818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.812783    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.814035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.814012    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.814418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.814400    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.829251 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.829220    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.829642 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.829620    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvdk\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:41.957930 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:41.957876    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:42.285736 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:50:42.285694    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0959bf2_4b14_4956_a695_fddffd81a9aa.slice/crio-f6762a7b77b1ce56f5467b9c3e0fc70edbe62568745ef3cf2749fba0bbf6c0f6 WatchSource:0}: Error finding container f6762a7b77b1ce56f5467b9c3e0fc70edbe62568745ef3cf2749fba0bbf6c0f6: Status 404 returned error can't find the container with id f6762a7b77b1ce56f5467b9c3e0fc70edbe62568745ef3cf2749fba0bbf6c0f6
Oct 29 22:50:42.288711 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.288679    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"]
Oct 29 22:50:42.290940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.290882    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:50:42.291048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.290991    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:50:42.291048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.291027    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:50:42.402643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.402609    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b" event={"ID":"770b4385-960b-4f68-9532-483d0dae622c","Type":"ContainerStarted","Data":"22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2"}
Oct 29 22:50:42.402810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.402677    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:42.402810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.402697    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b" event={"ID":"770b4385-960b-4f68-9532-483d0dae622c","Type":"ContainerStarted","Data":"cb758a8436753b0e68cae73b813b6dc5bc33a957067d5a53cdcc400920ecf930"}
Oct 29 22:50:42.404308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.404283    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" event={"ID":"f0959bf2-4b14-4956-a695-fddffd81a9aa","Type":"ContainerStarted","Data":"f6762a7b77b1ce56f5467b9c3e0fc70edbe62568745ef3cf2749fba0bbf6c0f6"}
Oct 29 22:50:42.457335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:42.457270    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b" podStartSLOduration=1.112663513 podStartE2EDuration="1.457248531s" totalImagesPullingTime="344.585018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:42.455634919 +0000 UTC m=+2182.398593621" watchObservedRunningTime="2026-10-29 22:50:42.457248531 +0000 UTC m=+2182.400207237"
Oct 29 22:50:43.422188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:43.422145    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" event={"ID":"f0959bf2-4b14-4956-a695-fddffd81a9aa","Type":"ContainerStarted","Data":"4bdea5ed81e4834dacbf2d3754fa611a2522b81e68fa723ce21eb93171c84d29"}
Oct 29 22:50:43.474596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:43.474546    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podStartSLOduration=2.474531967 podStartE2EDuration="2.474531967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:43.472777073 +0000 UTC m=+2183.415735779" watchObservedRunningTime="2026-10-29 22:50:43.474531967 +0000 UTC m=+2183.417490672"
Oct 29 22:50:43.958483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:43.958450    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:43.960388 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:43.960361    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:50:43.960513 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:43.960409    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:50:44.759731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.759704    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:50:44.870981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.870946    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871022    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871105    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871134    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871167    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871241    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.871360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871258    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871669 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871377    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871669 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871419    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.871669 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.871536    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket\") pod \"ef458bf1-c157-4388-b37a-ef8615d1b18a\" (UID: \"ef458bf1-c157-4388-b37a-ef8615d1b18a\") "
Oct 29 22:50:44.872133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.872112    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.872309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.872267    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.872533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.872510    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.872611 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.872473    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:44.872720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.872699    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.875203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.874786    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:44.875203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.875018    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "kube-api-access-bbr4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:44.875329 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.875285    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.876090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.876063    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token" pod "ef458bf1-c157-4388-b37a-ef8615d1b18a" (UID: "ef458bf1-c157-4388-b37a-ef8615d1b18a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:44.936576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.936549    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:50:44.959174 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.959137    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:50:44.959325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.959197    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:50:44.972983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.972954    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973191    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973234    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973390    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973444    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973495    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973539 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973531    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973595    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973722    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z\") pod \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\" (UID: \"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5\") "
Oct 29 22:50:44.973834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.973772    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.974095 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974070    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974384    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974478    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbr4d\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-kube-api-access-bbr4d\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974501    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974514    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974532    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974544    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974557    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974570    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef458bf1-c157-4388-b37a-ef8615d1b18a-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974582    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef458bf1-c157-4388-b37a-ef8615d1b18a-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974594    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974607    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef458bf1-c157-4388-b37a-ef8615d1b18a-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:44.975154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.974689    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:44.975789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.975280    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:44.976625 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.976586    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "kube-api-access-lqd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:44.979112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.979087    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:44.980182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.980153    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:44.981232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:44.981173    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy" pod "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" (UID: "cc9cd01f-2fd4-4f60-927f-7db36ca56aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:45.075760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075714    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.075760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075757    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.075994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075770    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.075994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075783    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqd9z\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-kube-api-access-lqd9z\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.075994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075801    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.075994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075813    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.075994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.075826    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.450130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.450093    4451 generic.go:356] "Generic (PLEG): container finished" podID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerID="4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4" exitCode=0
Oct 29 22:50:45.450320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.450161    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"
Oct 29 22:50:45.450320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.450191    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" event={"ID":"ef458bf1-c157-4388-b37a-ef8615d1b18a","Type":"ContainerDied","Data":"4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4"}
Oct 29 22:50:45.450320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.450245    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2" event={"ID":"ef458bf1-c157-4388-b37a-ef8615d1b18a","Type":"ContainerDied","Data":"eca87ed7c284567ba8790a9119f9ae5f3631a889bcc6acb8e0ce87b632195bd1"}
Oct 29 22:50:45.450320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.450267    4451 scope.go:122] "RemoveContainer" containerID="4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4"
Oct 29 22:50:45.456902 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.456857    4451 generic.go:356] "Generic (PLEG): container finished" podID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerID="fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce" exitCode=0
Oct 29 22:50:45.468088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.457528    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"
Oct 29 22:50:45.468088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.457754    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" event={"ID":"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5","Type":"ContainerDied","Data":"fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce"}
Oct 29 22:50:45.468088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.457797    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg" event={"ID":"cc9cd01f-2fd4-4f60-927f-7db36ca56aa5","Type":"ContainerDied","Data":"f8657df92a0792236ed37433bb069926a0883519c4a52570960e072ee7286f00"}
Oct 29 22:50:45.523816 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.523118    4451 scope.go:122] "RemoveContainer" containerID="4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4"
Oct 29 22:50:45.524080 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:45.524047    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4\": container with ID starting with 4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4 not found: ID does not exist" containerID="4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4"
Oct 29 22:50:45.524182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.524093    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4"} err="failed to get container status \"4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4\": rpc error: code = NotFound desc = could not find container \"4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4\": container with ID starting with 4c23e677609a17fff86ff2f5b229c933a7d62618c19de38345cd6b0747c65aa4 not found: ID does not exist"
Oct 29 22:50:45.524182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.524128    4451 scope.go:122] "RemoveContainer" containerID="fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce"
Oct 29 22:50:45.539861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.539830    4451 scope.go:122] "RemoveContainer" containerID="fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce"
Oct 29 22:50:45.540365 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:45.540339    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce\": container with ID starting with fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce not found: ID does not exist" containerID="fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce"
Oct 29 22:50:45.540495 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.540383    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce"} err="failed to get container status \"fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce\": rpc error: code = NotFound desc = could not find container \"fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce\": container with ID starting with fddbdce32ec7963c07177a4d0eee099393229f8933529566d2e0725113aab4ce not found: ID does not exist"
Oct 29 22:50:45.558991 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.558962    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"]
Oct 29 22:50:45.568590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.568564    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-h7dlg"]
Oct 29 22:50:45.622145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.622107    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"]
Oct 29 22:50:45.624149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.624123    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-v56n2"]
Oct 29 22:50:45.958369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.958336    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:50:45.958781 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:45.958385    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:50:46.164084 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:46.164052    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" path="/var/lib/kubelet/pods/cc9cd01f-2fd4-4f60-927f-7db36ca56aa5/volumes"
Oct 29 22:50:46.164623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:46.164606    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" path="/var/lib/kubelet/pods/ef458bf1-c157-4388-b37a-ef8615d1b18a/volumes"
Oct 29 22:50:46.862951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:46.859032    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"]
Oct 29 22:50:46.862951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:46.859356    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy" containerID="cri-o://149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685" gracePeriod=30
Oct 29 22:50:46.958865 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:46.958828    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:50:46.959336 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:46.958882    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:50:47.958852 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:47.958814    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:50:47.959061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:47.958870    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:50:48.428531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:48.428496    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 22:50:48.742884 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:48.742797    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:50:48.964183 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:48.964139    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:48.964747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:48.964700    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:48.966029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:48.966010    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 22:50:52.156759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.156726    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:50:52.308048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308014    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308048 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308052    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308082    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308317 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308187    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308352    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308355    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:52.308426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308366    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:52.308426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308393    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308445    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308459    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:52.308632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308508    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308582    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:52.308632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308624    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy\") pod \"331913b3-ed54-4d8f-866e-867d04f312b1\" (UID: \"331913b3-ed54-4d8f-866e-867d04f312b1\") "
Oct 29 22:50:52.308942 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.308723    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:52.309310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.309286    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331913b3-ed54-4d8f-866e-867d04f312b1-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.309485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.309468    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.309588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.309574    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.309684 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.309673    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.309772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.309759    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.312096 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.312039    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "kube-api-access-6fl2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:52.313866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.313830    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:52.314867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.314829    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:52.316794 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.316681    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy" pod "331913b3-ed54-4d8f-866e-867d04f312b1" (UID: "331913b3-ed54-4d8f-866e-867d04f312b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:52.411750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.411661    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331913b3-ed54-4d8f-866e-867d04f312b1-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.411750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.411691    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.411750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.411700    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331913b3-ed54-4d8f-866e-867d04f312b1-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.411750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.411709    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fl2c\" (UniqueName: \"kubernetes.io/projected/331913b3-ed54-4d8f-866e-867d04f312b1-kube-api-access-6fl2c\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:50:52.511341 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.511301    4451 generic.go:356] "Generic (PLEG): container finished" podID="331913b3-ed54-4d8f-866e-867d04f312b1" containerID="149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685" exitCode=0
Oct 29 22:50:52.511549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.511356    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"
Oct 29 22:50:52.511549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.511389    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" event={"ID":"331913b3-ed54-4d8f-866e-867d04f312b1","Type":"ContainerDied","Data":"149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685"}
Oct 29 22:50:52.511549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.511440    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m" event={"ID":"331913b3-ed54-4d8f-866e-867d04f312b1","Type":"ContainerDied","Data":"32e64d1959a3d2ba6949f94ab123da891fb34f548d1146099d40bfee620de22f"}
Oct 29 22:50:52.511549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.511474    4451 scope.go:122] "RemoveContainer" containerID="149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685"
Oct 29 22:50:52.529104 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.529076    4451 scope.go:122] "RemoveContainer" containerID="149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685"
Oct 29 22:50:52.529451 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:50:52.529424    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685\": container with ID starting with 149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685 not found: ID does not exist" containerID="149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685"
Oct 29 22:50:52.529566 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.529464    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685"} err="failed to get container status \"149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685\": rpc error: code = NotFound desc = could not find container \"149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685\": container with ID starting with 149086739b3f6ec2fee4b7d330fa5bcee7043d1dfdd5a2fe7825bb7994695685 not found: ID does not exist"
Oct 29 22:50:52.569577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.569546    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"]
Oct 29 22:50:52.572790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:52.572766    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2qx7m"]
Oct 29 22:50:54.076967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.076936    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"]
Oct 29 22:50:54.077578 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077562    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy"
Oct 29 22:50:54.077641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077579    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy"
Oct 29 22:50:54.077641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077613    4451 cpu_manager.go:414] "containerMap: removing container" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy"
Oct 29 22:50:54.077641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077621    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy"
Oct 29 22:50:54.077641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077634    4451 cpu_manager.go:414] "containerMap: removing container" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy"
Oct 29 22:50:54.077641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077639    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy"
Oct 29 22:50:54.077817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077748    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9cd01f-2fd4-4f60-927f-7db36ca56aa5" containerName="istio-proxy"
Oct 29 22:50:54.077817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077758    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" containerName="istio-proxy"
Oct 29 22:50:54.077817 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.077769    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef458bf1-c157-4388-b37a-ef8615d1b18a" containerName="istio-proxy"
Oct 29 22:50:54.081414 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.081393    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.099186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.099158    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"]
Oct 29 22:50:54.105590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.105562    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3bd0f6\"/\"openshift-service-ca.crt\""
Oct 29 22:50:54.105743 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.105616    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3bd0f6\"/\"kube-root-ca.crt\""
Oct 29 22:50:54.105743 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.105712    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3bd0f6\"/\"sub-test-model-3bd0f6-kserve-self-signed-certs\""
Oct 29 22:50:54.106049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.106029    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3bd0f6\"/\"default-dockercfg-h7pc4\""
Oct 29 22:50:54.134532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.134506    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.134715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.134551    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.134715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.134611    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.134715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.134660    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.135757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.135730    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.135879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.135864    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwbr\" (UniqueName: \"kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.166990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.166962    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331913b3-ed54-4d8f-866e-867d04f312b1" path="/var/lib/kubelet/pods/331913b3-ed54-4d8f-866e-867d04f312b1/volumes"
Oct 29 22:50:54.237595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.237561    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.237786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.237601    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.237786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.237666    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.237786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.237701    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.238022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.237859    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwbr\" (UniqueName: \"kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.238086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.238036    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.238138 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.238095    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.238204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.238137    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.238389 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.238360    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.240715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.240685    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.240844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.240822    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.257076 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.257045    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwbr\" (UniqueName: \"kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr\") pod \"sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") " pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.399263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.399155    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:50:54.733490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:54.733459    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"]
Oct 29 22:50:54.733923 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:50:54.733889    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe453da7_4c8a_4ba5_a09a_7362f7e0a499.slice/crio-2f7193f53e27c6b2258bb4009d36575d94b9355d1bda69628d02008d56b802b8 WatchSource:0}: Error finding container 2f7193f53e27c6b2258bb4009d36575d94b9355d1bda69628d02008d56b802b8: Status 404 returned error can't find the container with id 2f7193f53e27c6b2258bb4009d36575d94b9355d1bda69628d02008d56b802b8
Oct 29 22:50:55.539203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:55.539166    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd" event={"ID":"be453da7-4c8a-4ba5-a09a-7362f7e0a499","Type":"ContainerStarted","Data":"d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50"}
Oct 29 22:50:55.539610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:55.539209    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd" event={"ID":"be453da7-4c8a-4ba5-a09a-7362f7e0a499","Type":"ContainerStarted","Data":"2f7193f53e27c6b2258bb4009d36575d94b9355d1bda69628d02008d56b802b8"}
Oct 29 22:50:55.576067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:55.576019    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd" podStartSLOduration=1.576007103 podStartE2EDuration="1.576007103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:55.573373901 +0000 UTC m=+2195.516332608" watchObservedRunningTime="2026-10-29 22:50:55.576007103 +0000 UTC m=+2195.518965843"
Oct 29 22:50:56.400267 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:50:56.400231    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:51:03.439306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.439208    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"]
Oct 29 22:51:03.443728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.443705    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.460287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.460252    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"]
Oct 29 22:51:03.466624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.466599    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"openshift-service-ca.crt\""
Oct 29 22:51:03.466765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.466708    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"kube-root-ca.crt\""
Oct 29 22:51:03.477959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.470426    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"default-dockercfg-bhj4t\""
Oct 29 22:51:03.478078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.478018    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"auth-test-model-f8ca96-kserve-self-signed-certs\""
Oct 29 22:51:03.574905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.574864    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.575080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.575059    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.575124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.575097    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.575198 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.575180    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jsr\" (UniqueName: \"kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.575356 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.575336    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.575432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.575418    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.676830 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.676786    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.676873    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.677001    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.677304    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.677319    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677659 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.677634    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.677676    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.677754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.677725    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jsr\" (UniqueName: \"kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.679209 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.678253    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.680380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.680354    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.680508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.680413    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.704180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.696391    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jsr\" (UniqueName: \"kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr\") pod \"auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:03.780165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:03.780113    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:04.129339 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.129308    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"]
Oct 29 22:51:04.130468 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:51:04.130439    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f94dd79_def6_4aca_b33d_6b4cdab3e9b1.slice/crio-eda39133c2099d909f745e457e0608775944e53fc9a0d79bb86c0be34837d9f8 WatchSource:0}: Error finding container eda39133c2099d909f745e457e0608775944e53fc9a0d79bb86c0be34837d9f8: Status 404 returned error can't find the container with id eda39133c2099d909f745e457e0608775944e53fc9a0d79bb86c0be34837d9f8
Oct 29 22:51:04.400323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.400239    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:51:04.412993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.412966    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:51:04.605409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.605372    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" event={"ID":"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1","Type":"ContainerStarted","Data":"f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b"}
Oct 29 22:51:04.605849 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.605423    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" event={"ID":"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1","Type":"ContainerStarted","Data":"eda39133c2099d909f745e457e0608775944e53fc9a0d79bb86c0be34837d9f8"}
Oct 29 22:51:04.616678 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.616649    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:51:04.645629 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.645578    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" podStartSLOduration=1.6455654530000001 podStartE2EDuration="1.645565453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:04.642538889 +0000 UTC m=+2204.585497595" watchObservedRunningTime="2026-10-29 22:51:04.645565453 +0000 UTC m=+2204.588524191"
Oct 29 22:51:04.780832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:04.780756    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:13.781124 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:13.781034    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:13.798804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:13.798773    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:14.696333 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:14.696301    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:51:18.742336 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:51:18.742296    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:51:24.670677 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.670640    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"]
Oct 29 22:51:24.680363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.680336    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.698057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.697956    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"]
Oct 29 22:51:24.707650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.707616    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-db7419\"/\"openshift-service-ca.crt\""
Oct 29 22:51:24.708366 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.708271    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-db7419\"/\"default-dockercfg-s2w9m\""
Oct 29 22:51:24.708633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.708615    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-db7419\"/\"auth-test-model-db7419-kserve-self-signed-certs\""
Oct 29 22:51:24.708849 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.708834    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-db7419\"/\"kube-root-ca.crt\""
Oct 29 22:51:24.823244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.823202    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.823441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.823397    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.823441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.823421    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.823566 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.823447    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdgv\" (UniqueName: \"kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.823566 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.823541    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.823566 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.823562    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.924460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.924372    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.924460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.924413    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.924725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.924475    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.924725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.924669    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.924725 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.924702    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.924893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.924740    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdgv\" (UniqueName: \"kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.925218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.925193    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.925524 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.925500    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.925524 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.925515    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.927811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.927786    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.927996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.927810    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:24.949810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:24.949763    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdgv\" (UniqueName: \"kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv\") pod \"auth-test-model-db7419-kserve-5456b56689-c6jz9\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") " pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:25.085583 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:25.085547    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:25.460958 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:25.459899    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"]
Oct 29 22:51:25.786486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:25.786389    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9" event={"ID":"8cae6056-f684-468a-8efb-77a1e6876396","Type":"ContainerStarted","Data":"619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f"}
Oct 29 22:51:25.786486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:25.786440    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9" event={"ID":"8cae6056-f684-468a-8efb-77a1e6876396","Type":"ContainerStarted","Data":"f46bcffa9bad7783288d8afa59723e2949afdc96b103dc1814b9f0b9d5b945e0"}
Oct 29 22:51:25.821576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:25.821509    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9" podStartSLOduration=1.821492059 podStartE2EDuration="1.821492059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:25.819771152 +0000 UTC m=+2225.762729859" watchObservedRunningTime="2026-10-29 22:51:25.821492059 +0000 UTC m=+2225.764450764"
Oct 29 22:51:26.085672 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:26.085634    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:35.085939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:35.085885    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:35.098873 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:35.098842    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:35.861272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:35.861240    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:51:38.602742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:38.602703    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"]
Oct 29 22:51:38.603148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:38.602935    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy" containerID="cri-o://5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299" gracePeriod=30
Oct 29 22:51:40.294098 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:40.294062    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"]
Oct 29 22:51:43.880883 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.880826    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:51:43.905308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.905276    4451 generic.go:356] "Generic (PLEG): container finished" podID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerID="5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299" exitCode=0
Oct 29 22:51:43.905456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.905327    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"
Oct 29 22:51:43.905456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.905353    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" event={"ID":"4d9ef1bd-4959-469e-9b12-4df1f39de316","Type":"ContainerDied","Data":"5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299"}
Oct 29 22:51:43.905456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.905409    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf" event={"ID":"4d9ef1bd-4959-469e-9b12-4df1f39de316","Type":"ContainerDied","Data":"e6174f5f92ec9d0091b1f8f74e1c33e2ec9dc7c31cf0256add898a4de27f345d"}
Oct 29 22:51:43.905456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.905432    4451 scope.go:122] "RemoveContainer" containerID="5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299"
Oct 29 22:51:43.915482 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.915453    4451 scope.go:122] "RemoveContainer" containerID="5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299"
Oct 29 22:51:43.915735 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:51:43.915715    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299\": container with ID starting with 5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299 not found: ID does not exist" containerID="5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299"
Oct 29 22:51:43.915800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:43.915746    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299"} err="failed to get container status \"5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299\": rpc error: code = NotFound desc = could not find container \"5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299\": container with ID starting with 5e59e493fdc727139979ed09dfc5ad5a3d7fe45f72eb8587819eda7fd61bc299 not found: ID does not exist"
Oct 29 22:51:44.001380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001347    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.001541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001390    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.001541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001419    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.001541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001461    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.001541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001530    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.001796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001565    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.001796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001652    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:44.001796 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001737    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.002000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001838    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.002000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001853    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:44.002000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001869    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo\") pod \"4d9ef1bd-4959-469e-9b12-4df1f39de316\" (UID: \"4d9ef1bd-4959-469e-9b12-4df1f39de316\") "
Oct 29 22:51:44.002000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001887    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:44.002000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001898    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:51:44.002000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.001902    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:44.002637 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.002618    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.002637 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.002637    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9ef1bd-4959-469e-9b12-4df1f39de316-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.002759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.002646    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.002759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.002656    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.002759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.002663    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.004067 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.004039    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "kube-api-access-mb5l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:44.004157 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.004071    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:44.004335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.004312    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:44.004428 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.004324    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo" pod "4d9ef1bd-4959-469e-9b12-4df1f39de316" (UID: "4d9ef1bd-4959-469e-9b12-4df1f39de316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:51:44.105312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.105272    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.105312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.105303    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb5l6\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-kube-api-access-mb5l6\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.105312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.105314    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.105312 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.105324    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9ef1bd-4959-469e-9b12-4df1f39de316-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:44.242389 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.242357    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"]
Oct 29 22:51:44.252714 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:44.252684    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-9txdf"]
Oct 29 22:51:46.170252 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:46.170217    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" path="/var/lib/kubelet/pods/4d9ef1bd-4959-469e-9b12-4df1f39de316/volumes"
Oct 29 22:51:46.184842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:46.184813    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"]
Oct 29 22:51:48.754359 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:51:48.754276    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:51:51.974729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:51.974693    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"]
Oct 29 22:51:51.975569 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:51.975545    4451 cpu_manager.go:414] "containerMap: removing container" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy"
Oct 29 22:51:51.975682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:51.975570    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy"
Oct 29 22:51:51.975761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:51.975746    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9ef1bd-4959-469e-9b12-4df1f39de316" containerName="istio-proxy"
Oct 29 22:51:51.979453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:51.979432    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.007753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.007724    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"]
Oct 29 22:51:52.021308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.021287    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-02720a-serving-cert\""
Oct 29 22:51:52.124644 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.124606    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.124808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.124679    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzh4\" (UniqueName: \"kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.124808 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.124792    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.124961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.124930    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.226070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.225964    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzh4\" (UniqueName: \"kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.226366 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.226335    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.226502 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.226484    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.227149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.226674    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.227772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.227737    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.229806 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.229780    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.230223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.230193    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.244893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.244862    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzh4\" (UniqueName: \"kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4\") pod \"maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.293826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.293791    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:52.600841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.600811    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"]
Oct 29 22:51:52.601433 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:51:52.601406    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe906733_a9c7_4738_99ee_40d41b4824a2.slice/crio-b93b00bd070f9c96cd75991e8504757ee1fa54ced6a0c5a9f3d9aae811111037 WatchSource:0}: Error finding container b93b00bd070f9c96cd75991e8504757ee1fa54ced6a0c5a9f3d9aae811111037: Status 404 returned error can't find the container with id b93b00bd070f9c96cd75991e8504757ee1fa54ced6a0c5a9f3d9aae811111037
Oct 29 22:51:52.973833 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:52.973748    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5" event={"ID":"be906733-a9c7-4738-99ee-40d41b4824a2","Type":"ContainerStarted","Data":"b93b00bd070f9c96cd75991e8504757ee1fa54ced6a0c5a9f3d9aae811111037"}
Oct 29 22:51:53.985614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:53.985567    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5" event={"ID":"be906733-a9c7-4738-99ee-40d41b4824a2","Type":"ContainerStarted","Data":"d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444"}
Oct 29 22:51:53.986053 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:53.985758    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:51:54.028308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:54.028233    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5" podStartSLOduration=2.620071556 podStartE2EDuration="3.02821349s" totalImagesPullingTime="408.141934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:54.026978232 +0000 UTC m=+2253.969936944" watchObservedRunningTime="2026-10-29 22:51:54.02821349 +0000 UTC m=+2253.971172196"
Oct 29 22:51:55.335888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.335849    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" podUID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" containerName="main" containerID="cri-o://a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d" gracePeriod=45
Oct 29 22:51:55.354742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.354713    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" podUID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": dial tcp 10.133.0.114:8000: connect: connection refused"
Oct 29 22:51:55.642045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.642016    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:51:55.786540 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786504    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home\") pod \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") "
Oct 29 22:51:55.786722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786592    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm\") pod \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") "
Oct 29 22:51:55.786722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786620    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs\") pod \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") "
Oct 29 22:51:55.786722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786706    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir\") pod \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") "
Oct 29 22:51:55.786988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786746    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache\") pod \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") "
Oct 29 22:51:55.786988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786771    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home" pod "7e38a10c-9fbf-478e-9e7c-0951f44eb630" (UID: "7e38a10c-9fbf-478e-9e7c-0951f44eb630"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:55.786988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.786800    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw\" (UniqueName: \"kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw\") pod \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\" (UID: \"7e38a10c-9fbf-478e-9e7c-0951f44eb630\") "
Oct 29 22:51:55.787200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.787028    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache" pod "7e38a10c-9fbf-478e-9e7c-0951f44eb630" (UID: "7e38a10c-9fbf-478e-9e7c-0951f44eb630"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:55.787262 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.787205    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir" pod "7e38a10c-9fbf-478e-9e7c-0951f44eb630" (UID: "7e38a10c-9fbf-478e-9e7c-0951f44eb630"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:55.787426 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.787408    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:55.787622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.787428    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:55.787622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.787441    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:55.789268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.789242    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw" pod "7e38a10c-9fbf-478e-9e7c-0951f44eb630" (UID: "7e38a10c-9fbf-478e-9e7c-0951f44eb630"). InnerVolumeSpecName "kube-api-access-rl7mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:55.789367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.789317    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs" pod "7e38a10c-9fbf-478e-9e7c-0951f44eb630" (UID: "7e38a10c-9fbf-478e-9e7c-0951f44eb630"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:55.789367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.789327    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm" pod "7e38a10c-9fbf-478e-9e7c-0951f44eb630" (UID: "7e38a10c-9fbf-478e-9e7c-0951f44eb630"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:51:55.888858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.888783    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e38a10c-9fbf-478e-9e7c-0951f44eb630-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:55.888858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.888813    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e38a10c-9fbf-478e-9e7c-0951f44eb630-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:55.888858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.888824    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl7mw\" (UniqueName: \"kubernetes.io/projected/7e38a10c-9fbf-478e-9e7c-0951f44eb630-kube-api-access-rl7mw\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:51:55.999889 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.999856    4451 generic.go:356] "Generic (PLEG): container finished" podID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" containerID="a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d" exitCode=0
Oct 29 22:51:56.000092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.999899    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"
Oct 29 22:51:56.000092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.999947    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" event={"ID":"7e38a10c-9fbf-478e-9e7c-0951f44eb630","Type":"ContainerDied","Data":"a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d"}
Oct 29 22:51:56.000092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:55.999995    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w" event={"ID":"7e38a10c-9fbf-478e-9e7c-0951f44eb630","Type":"ContainerDied","Data":"1cddf9a03c318ae85f19f215c51825d0da6f698c6047a1bd0517315f28133adb"}
Oct 29 22:51:56.000092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:56.000017    4451 scope.go:122] "RemoveContainer" containerID="a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d"
Oct 29 22:51:56.011189 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:56.011170    4451 scope.go:122] "RemoveContainer" containerID="a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d"
Oct 29 22:51:56.011441 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:51:56.011417    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d\": container with ID starting with a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d not found: ID does not exist" containerID="a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d"
Oct 29 22:51:56.011503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:56.011454    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d"} err="failed to get container status \"a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d\": rpc error: code = NotFound desc = could not find container \"a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d\": container with ID starting with a2f4afc4588b2edfa854e27a92653a65881d06550d6ac1568ab83910010b5b1d not found: ID does not exist"
Oct 29 22:51:56.046745 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:56.046720    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"]
Oct 29 22:51:56.049892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:56.049871    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-06a95f/auth-test-model-06a95f-kserve-5db98dbc57-xpl7w"]
Oct 29 22:51:56.163519 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:56.163447    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" path="/var/lib/kubelet/pods/7e38a10c-9fbf-478e-9e7c-0951f44eb630/volumes"
Oct 29 22:51:59.999565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:51:59.999534    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 22:52:01.214391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.214326    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9" podUID="8cae6056-f684-468a-8efb-77a1e6876396" containerName="main" containerID="cri-o://619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f" gracePeriod=45
Oct 29 22:52:01.526012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.525981    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579159    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs\") pod \"8cae6056-f684-468a-8efb-77a1e6876396\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") "
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579247    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm\") pod \"8cae6056-f684-468a-8efb-77a1e6876396\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") "
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579291    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home\") pod \"8cae6056-f684-468a-8efb-77a1e6876396\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") "
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579347    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache\") pod \"8cae6056-f684-468a-8efb-77a1e6876396\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") "
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579438    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv\" (UniqueName: \"kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv\") pod \"8cae6056-f684-468a-8efb-77a1e6876396\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") "
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579534    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir\") pod \"8cae6056-f684-468a-8efb-77a1e6876396\" (UID: \"8cae6056-f684-468a-8efb-77a1e6876396\") "
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.579568    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home" pod "8cae6056-f684-468a-8efb-77a1e6876396" (UID: "8cae6056-f684-468a-8efb-77a1e6876396"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:01.580263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.580229    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:01.580869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.580316    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir" pod "8cae6056-f684-468a-8efb-77a1e6876396" (UID: "8cae6056-f684-468a-8efb-77a1e6876396"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:01.580869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.580428    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache" pod "8cae6056-f684-468a-8efb-77a1e6876396" (UID: "8cae6056-f684-468a-8efb-77a1e6876396"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:01.581701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.581670    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs" pod "8cae6056-f684-468a-8efb-77a1e6876396" (UID: "8cae6056-f684-468a-8efb-77a1e6876396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:01.582052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.582028    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv" pod "8cae6056-f684-468a-8efb-77a1e6876396" (UID: "8cae6056-f684-468a-8efb-77a1e6876396"). InnerVolumeSpecName "kube-api-access-wgdgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:01.582191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.582087    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm" pod "8cae6056-f684-468a-8efb-77a1e6876396" (UID: "8cae6056-f684-468a-8efb-77a1e6876396"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:01.681484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.681452    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:01.681484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.681479    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:01.681484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.681490    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdgv\" (UniqueName: \"kubernetes.io/projected/8cae6056-f684-468a-8efb-77a1e6876396-kube-api-access-wgdgv\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:01.681714 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.681499    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cae6056-f684-468a-8efb-77a1e6876396-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:01.681714 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:01.681509    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cae6056-f684-468a-8efb-77a1e6876396-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:02.046350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.046314    4451 generic.go:356] "Generic (PLEG): container finished" podID="8cae6056-f684-468a-8efb-77a1e6876396" containerID="619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f" exitCode=0
Oct 29 22:52:02.046561 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.046369    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"
Oct 29 22:52:02.046561 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.046407    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9" event={"ID":"8cae6056-f684-468a-8efb-77a1e6876396","Type":"ContainerDied","Data":"619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f"}
Oct 29 22:52:02.046561 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.046452    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9" event={"ID":"8cae6056-f684-468a-8efb-77a1e6876396","Type":"ContainerDied","Data":"f46bcffa9bad7783288d8afa59723e2949afdc96b103dc1814b9f0b9d5b945e0"}
Oct 29 22:52:02.046561 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.046476    4451 scope.go:122] "RemoveContainer" containerID="619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f"
Oct 29 22:52:02.058791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.058772    4451 scope.go:122] "RemoveContainer" containerID="619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f"
Oct 29 22:52:02.059153 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:02.059133    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f\": container with ID starting with 619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f not found: ID does not exist" containerID="619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f"
Oct 29 22:52:02.059216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.059163    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f"} err="failed to get container status \"619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f\": rpc error: code = NotFound desc = could not find container \"619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f\": container with ID starting with 619403151f6c45fd492c9f2832396939846b60f8de58728620f2b380d2d9b92f not found: ID does not exist"
Oct 29 22:52:02.092641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.092618    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"]
Oct 29 22:52:02.102131 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.102107    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-db7419/auth-test-model-db7419-kserve-5456b56689-c6jz9"]
Oct 29 22:52:02.168712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:02.168672    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cae6056-f684-468a-8efb-77a1e6876396" path="/var/lib/kubelet/pods/8cae6056-f684-468a-8efb-77a1e6876396/volumes"
Oct 29 22:52:06.647159 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.647120    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"]
Oct 29 22:52:06.648548 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.648518    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" containerName="main"
Oct 29 22:52:06.648754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.648735    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" containerName="main"
Oct 29 22:52:06.648994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.648963    4451 cpu_manager.go:414] "containerMap: removing container" podUID="8cae6056-f684-468a-8efb-77a1e6876396" containerName="main"
Oct 29 22:52:06.649142 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.649114    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cae6056-f684-468a-8efb-77a1e6876396" containerName="main"
Oct 29 22:52:06.649661 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.649641    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e38a10c-9fbf-478e-9e7c-0951f44eb630" containerName="main"
Oct 29 22:52:06.649791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.649780    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cae6056-f684-468a-8efb-77a1e6876396" containerName="main"
Oct 29 22:52:06.657507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.657481    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.664748 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.664723    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"]
Oct 29 22:52:06.673292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.671979    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3ee017-serving-cert\""
Oct 29 22:52:06.757039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.757007    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.757200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.757126    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.757200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.757174    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wml9\" (UniqueName: \"kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.757362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.757337    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.858435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.858405    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.858602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.858501    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.858602 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.858562    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.858686 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.858606    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wml9\" (UniqueName: \"kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.859357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.859315    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.861043 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.861013    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.861133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.861122    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.875501 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.875480    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wml9\" (UniqueName: \"kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9\") pod \"maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:06.977387 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:06.977305    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:07.283719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:07.283689    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"]
Oct 29 22:52:07.284999 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:07.284974    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda038013b_5758_420e_a8f0_05ba413a1fcb.slice/crio-5ab1f7f7e78944f7ab88337fe98a5dfcafbd8cf5307677e56105656532b382f9 WatchSource:0}: Error finding container 5ab1f7f7e78944f7ab88337fe98a5dfcafbd8cf5307677e56105656532b382f9: Status 404 returned error can't find the container with id 5ab1f7f7e78944f7ab88337fe98a5dfcafbd8cf5307677e56105656532b382f9
Oct 29 22:52:08.087595 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:08.087556    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx" event={"ID":"a038013b-5758-420e-a8f0-05ba413a1fcb","Type":"ContainerStarted","Data":"b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa"}
Oct 29 22:52:08.088263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:08.087615    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx" event={"ID":"a038013b-5758-420e-a8f0-05ba413a1fcb","Type":"ContainerStarted","Data":"5ab1f7f7e78944f7ab88337fe98a5dfcafbd8cf5307677e56105656532b382f9"}
Oct 29 22:52:08.088263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:08.087641    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:08.127881 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:08.127834    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx" podStartSLOduration=1.766692393 podStartE2EDuration="2.127819298s" totalImagesPullingTime="361.126905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:08.124285337 +0000 UTC m=+2268.067244040" watchObservedRunningTime="2026-10-29 22:52:08.127819298 +0000 UTC m=+2268.070778050"
Oct 29 22:52:09.427671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.427638    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"]
Oct 29 22:52:09.433581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.433547    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.461947 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.461883    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"]
Oct 29 22:52:09.534363 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.534332    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f8ca96-openshift-default-dockercfg-ppssn\""
Oct 29 22:52:09.598696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.598659    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.598879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.598699    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.598879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.598730    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqc5f\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.599025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.598931    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.599025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.598985    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.599114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.599088    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.599158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.599129    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.599236 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.599223    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.599272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.599247    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700325    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700373    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700461    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700490    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700525    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqc5f\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700617    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700661    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700772    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700835    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.700882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700853    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.701064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.700890    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.701287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.701260    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.701287 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.701284    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.701496 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.701300    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.703074 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.703047    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.703167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.703124    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.718037 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.718013    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.718283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.718267    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqc5f\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:09.765506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:09.765482    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:10.094323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.094285    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"]
Oct 29 22:52:10.109948 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:10.109878    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ccc1a0_98cb_4366_bb9b_6bca93a999a1.slice/crio-fc0883e091f9cd7e933d863f3a4ca0594ff9f238aab125f73bbc5a488592036c WatchSource:0}: Error finding container fc0883e091f9cd7e933d863f3a4ca0594ff9f238aab125f73bbc5a488592036c: Status 404 returned error can't find the container with id fc0883e091f9cd7e933d863f3a4ca0594ff9f238aab125f73bbc5a488592036c
Oct 29 22:52:10.118264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.116230    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"]
Oct 29 22:52:10.118264 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.116366    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.119172 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.119120    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"]
Oct 29 22:52:10.121099 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.121054    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:10.121192 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.121137    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:10.121192 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.121179    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:10.133123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.130725    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"default-dockercfg-88m6v\""
Oct 29 22:52:10.133123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.130975    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"auth-test-model-c933cb-kserve-self-signed-certs\""
Oct 29 22:52:10.133123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.131180    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"openshift-service-ca.crt\""
Oct 29 22:52:10.137225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.135309    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"kube-root-ca.crt\""
Oct 29 22:52:10.310935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.310840    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.311123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.310973    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.311123 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.311037    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.311239 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.311202    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.311303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.311237    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.311422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.311398    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ksz\" (UniqueName: \"kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.413783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.413672    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.413783 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.413758    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414042 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.413876    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ksz\" (UniqueName: \"kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.414097    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.414142    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414284 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.414171    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.414554    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414678 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.414601    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.414738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.414693    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.417347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.417318    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.417997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.417962    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.431165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.431144    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ksz\" (UniqueName: \"kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz\") pod \"auth-test-model-c933cb-kserve-659668974f-jtwb6\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.443753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.443729    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:10.756573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:10.756544    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"]
Oct 29 22:52:10.757802 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:10.757774    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d404b47_2069_48ef_9a0d_1404f4feaff2.slice/crio-4971c9aeaffd5b4d10329e48a12881dafa6fede4be0dc4317762ae30017884f7 WatchSource:0}: Error finding container 4971c9aeaffd5b4d10329e48a12881dafa6fede4be0dc4317762ae30017884f7: Status 404 returned error can't find the container with id 4971c9aeaffd5b4d10329e48a12881dafa6fede4be0dc4317762ae30017884f7
Oct 29 22:52:11.122323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.121104    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" event={"ID":"37ccc1a0-98cb-4366-bb9b-6bca93a999a1","Type":"ContainerStarted","Data":"7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e"}
Oct 29 22:52:11.122323 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.121181    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" event={"ID":"37ccc1a0-98cb-4366-bb9b-6bca93a999a1","Type":"ContainerStarted","Data":"fc0883e091f9cd7e933d863f3a4ca0594ff9f238aab125f73bbc5a488592036c"}
Oct 29 22:52:11.124721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.124686    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6" event={"ID":"6d404b47-2069-48ef-9a0d-1404f4feaff2","Type":"ContainerStarted","Data":"5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de"}
Oct 29 22:52:11.124884 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.124741    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6" event={"ID":"6d404b47-2069-48ef-9a0d-1404f4feaff2","Type":"ContainerStarted","Data":"4971c9aeaffd5b4d10329e48a12881dafa6fede4be0dc4317762ae30017884f7"}
Oct 29 22:52:11.183133 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.183085    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podStartSLOduration=2.183071536 podStartE2EDuration="2.183071536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:11.170264674 +0000 UTC m=+2271.113223380" watchObservedRunningTime="2026-10-29 22:52:11.183071536 +0000 UTC m=+2271.126030240"
Oct 29 22:52:11.219532 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.219464    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6" podStartSLOduration=1.219443638 podStartE2EDuration="1.219443638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:11.219273787 +0000 UTC m=+2271.162232496" watchObservedRunningTime="2026-10-29 22:52:11.219443638 +0000 UTC m=+2271.162402344"
Oct 29 22:52:11.444172 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.444089    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:11.766411 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.766330    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:11.767977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.767944    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:52:11.768047 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:11.768008    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 29 22:52:12.772107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:12.772058    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:52:12.772609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:12.772120    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 29 22:52:13.766421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:13.766329    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:52:13.766606 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:13.766409    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 29 22:52:14.119008 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:14.118948    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 22:52:14.765896 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:14.765858    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:52:14.766101 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:14.765952    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 29 22:52:15.766339 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:15.766299    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:52:15.766727 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:15.766363    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 29 22:52:16.766465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:16.766424    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:52:16.766967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:16.766488    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 29 22:52:17.777144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:17.777110    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:17.777703 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:17.777642    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:17.778441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:17.778417    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 22:52:18.742054 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:18.742015    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:52:20.444647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:20.444614    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:20.457379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:20.457352    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:21.210155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:21.210123    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:52:31.168896 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.168859    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"]
Oct 29 22:52:31.174726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.174706    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.190410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.190329    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"]
Oct 29 22:52:31.197379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.196771    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c87bf1-serving-cert\""
Oct 29 22:52:31.225549 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.225511    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.225730 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.225588    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.225730 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.225717    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdlz\" (UniqueName: \"kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.225847 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.225761    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.327597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.327559    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdlz\" (UniqueName: \"kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.327804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.327625    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.327804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.327676    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.327804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.327731    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.328472 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.328444    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.330243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.330217    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.330395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.330374    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.345104 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.345083    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdlz\" (UniqueName: \"kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz\") pod \"maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.489319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.489229    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:31.797519 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:31.797485    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"]
Oct 29 22:52:31.797631 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:31.797608    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3494f4d4_87e6_4d73_88b2_2a32d6db7554.slice/crio-928377aaf81b74e69d7e2244fbad1f4a8b1804c9302cca5b35bed4d3b95024be WatchSource:0}: Error finding container 928377aaf81b74e69d7e2244fbad1f4a8b1804c9302cca5b35bed4d3b95024be: Status 404 returned error can't find the container with id 928377aaf81b74e69d7e2244fbad1f4a8b1804c9302cca5b35bed4d3b95024be
Oct 29 22:52:32.271297 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:32.271256    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9" event={"ID":"3494f4d4-87e6-4d73-88b2-2a32d6db7554","Type":"ContainerStarted","Data":"a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed"}
Oct 29 22:52:32.271799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:32.271332    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:32.271799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:32.271357    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9" event={"ID":"3494f4d4-87e6-4d73-88b2-2a32d6db7554","Type":"ContainerStarted","Data":"928377aaf81b74e69d7e2244fbad1f4a8b1804c9302cca5b35bed4d3b95024be"}
Oct 29 22:52:32.312246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:32.312196    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9" podStartSLOduration=1.012845915 podStartE2EDuration="1.312169461s" totalImagesPullingTime="299.323546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:32.311263076 +0000 UTC m=+2292.254221781" watchObservedRunningTime="2026-10-29 22:52:32.312169461 +0000 UTC m=+2292.255128167"
Oct 29 22:52:38.282429 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:38.282344    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:52:48.761301 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:48.761237    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:52:49.448754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.448720    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-d4316c"]
Oct 29 22:52:49.453003 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.452984    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:49.474825 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.474441    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:52:49.474997 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.474905    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-d4316c"]
Oct 29 22:52:49.530335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.530301    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwl2x\" (UniqueName: \"kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x\") pod \"test-curl-9092-d4316c\" (UID: \"3a0a3325-9385-4384-b736-22139499892b\") " pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:49.631641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.631607    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwl2x\" (UniqueName: \"kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x\") pod \"test-curl-9092-d4316c\" (UID: \"3a0a3325-9385-4384-b736-22139499892b\") " pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:49.649413 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.649384    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwl2x\" (UniqueName: \"kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x\") pod \"test-curl-9092-d4316c\" (UID: \"3a0a3325-9385-4384-b736-22139499892b\") " pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:49.770633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:49.770558    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:50.066112 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:50.066079    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-d4316c"]
Oct 29 22:52:50.067624 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:50.067594    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a0a3325_9385_4384_b736_22139499892b.slice/crio-41f2de24d24d6d88b4c4ea8c7f4f2a7c825a893bf2967a5480b2d976b145b90a WatchSource:0}: Error finding container 41f2de24d24d6d88b4c4ea8c7f4f2a7c825a893bf2967a5480b2d976b145b90a: Status 404 returned error can't find the container with id 41f2de24d24d6d88b4c4ea8c7f4f2a7c825a893bf2967a5480b2d976b145b90a
Oct 29 22:52:50.386615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:50.386565    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-d4316c" event={"ID":"3a0a3325-9385-4384-b736-22139499892b","Type":"ContainerStarted","Data":"acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828"}
Oct 29 22:52:50.386772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:50.386621    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-d4316c" event={"ID":"3a0a3325-9385-4384-b736-22139499892b","Type":"ContainerStarted","Data":"41f2de24d24d6d88b4c4ea8c7f4f2a7c825a893bf2967a5480b2d976b145b90a"}
Oct 29 22:52:50.426600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:50.426551    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-d4316c" podStartSLOduration=1.426539392 podStartE2EDuration="1.426539392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:50.423937806 +0000 UTC m=+2310.366896508" watchObservedRunningTime="2026-10-29 22:52:50.426539392 +0000 UTC m=+2310.369498096"
Oct 29 22:52:51.245069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.244131    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-d4316c"]
Oct 29 22:52:51.249983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.249954    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-d4316c"]
Oct 29 22:52:51.391070 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.391033    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-72e0f7"]
Oct 29 22:52:51.391988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.391958    4451 cpu_manager.go:414] "containerMap: removing container" podUID="3a0a3325-9385-4384-b736-22139499892b" containerName="test-curl-9092-d4316c"
Oct 29 22:52:51.391988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.391987    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0a3325-9385-4384-b736-22139499892b" containerName="test-curl-9092-d4316c"
Oct 29 22:52:51.392183 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.392124    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0a3325-9385-4384-b736-22139499892b" containerName="test-curl-9092-d4316c"
Oct 29 22:52:51.396035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.396013    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:51.411851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.411822    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-72e0f7"]
Oct 29 22:52:51.462554 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.462526    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bpr\" (UniqueName: \"kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr\") pod \"test-curl-9092-72e0f7\" (UID: \"50264019-541f-4668-b722-dda4db1ec046\") " pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:51.563959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.563823    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bpr\" (UniqueName: \"kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr\") pod \"test-curl-9092-72e0f7\" (UID: \"50264019-541f-4668-b722-dda4db1ec046\") " pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:51.581318 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.581284    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bpr\" (UniqueName: \"kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr\") pod \"test-curl-9092-72e0f7\" (UID: \"50264019-541f-4668-b722-dda4db1ec046\") " pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:51.714126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:51.714090    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:52.013036 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:52.013005    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-72e0f7"]
Oct 29 22:52:52.013444 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:52.013417    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50264019_541f_4668_b722_dda4db1ec046.slice/crio-7fb5bad1a218888c2d5dde2da27cdd91e7c8f1bd0dd06cf8b6a22bf156a71c2a WatchSource:0}: Error finding container 7fb5bad1a218888c2d5dde2da27cdd91e7c8f1bd0dd06cf8b6a22bf156a71c2a: Status 404 returned error can't find the container with id 7fb5bad1a218888c2d5dde2da27cdd91e7c8f1bd0dd06cf8b6a22bf156a71c2a
Oct 29 22:52:52.400769 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:52.400736    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-72e0f7" event={"ID":"50264019-541f-4668-b722-dda4db1ec046","Type":"ContainerStarted","Data":"cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0"}
Oct 29 22:52:52.400769 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:52.400776    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-72e0f7" event={"ID":"50264019-541f-4668-b722-dda4db1ec046","Type":"ContainerStarted","Data":"7fb5bad1a218888c2d5dde2da27cdd91e7c8f1bd0dd06cf8b6a22bf156a71c2a"}
Oct 29 22:52:52.401238 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:52.401003    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-d4316c" podUID="3a0a3325-9385-4384-b736-22139499892b" containerName="test-curl-9092-d4316c" containerID="cri-o://acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828" gracePeriod=2
Oct 29 22:52:52.434614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:52.434550    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-72e0f7" podStartSLOduration=1.434532926 podStartE2EDuration="1.434532926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:52.432425623 +0000 UTC m=+2312.375384324" watchObservedRunningTime="2026-10-29 22:52:52.434532926 +0000 UTC m=+2312.377491634"
Oct 29 22:52:53.163012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:53.162979    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-72e0f7"]
Oct 29 22:52:53.167360 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:53.167328    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-72e0f7"]
Oct 29 22:52:54.422393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.422359    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-72e0f7" podUID="50264019-541f-4668-b722-dda4db1ec046" containerName="test-curl-9092-72e0f7" containerID="cri-o://cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0" gracePeriod=2
Oct 29 22:52:54.548107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.548072    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"]
Oct 29 22:52:54.548715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.548691    4451 cpu_manager.go:414] "containerMap: removing container" podUID="50264019-541f-4668-b722-dda4db1ec046" containerName="test-curl-9092-72e0f7"
Oct 29 22:52:54.548715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.548707    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50264019-541f-4668-b722-dda4db1ec046" containerName="test-curl-9092-72e0f7"
Oct 29 22:52:54.548979 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.548848    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="50264019-541f-4668-b722-dda4db1ec046" containerName="test-curl-9092-72e0f7"
Oct 29 22:52:54.553470 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.553445    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.573489 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.573456    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"]
Oct 29 22:52:54.617555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.617525    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:54.709216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709145    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x\" (UniqueName: \"kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x\") pod \"3a0a3325-9385-4384-b736-22139499892b\" (UID: \"3a0a3325-9385-4384-b736-22139499892b\") "
Oct 29 22:52:54.709415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709394    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709568 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709548    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709584    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709699 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709670    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709718    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsrk\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709797    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709873    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.709982 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.709969    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.710052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.710027    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.711432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.711407    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x" pod "3a0a3325-9385-4384-b736-22139499892b" (UID: "3a0a3325-9385-4384-b736-22139499892b"). InnerVolumeSpecName "kube-api-access-xwl2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:54.811246 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811212    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811263    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811380    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsrk\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811424    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811466    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811512    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811541 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811534    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811628    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811687    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811717    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811846    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.811961 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.811868    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.812102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.812012    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwl2x\" (UniqueName: \"kubernetes.io/projected/3a0a3325-9385-4384-b736-22139499892b-kube-api-access-xwl2x\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:54.812102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.812058    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.812377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.812359    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.813987 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.813965    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.814256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.814239    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.828702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.828681    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.828814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.828685    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsrk\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:54.880721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:54.880693    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:55.201459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.201429    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"]
Oct 29 22:52:55.203118 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:52:55.203074    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6a29c6_375d_40d5_a2f2_144bc504e51f.slice/crio-073d7b2a5107fe6ac95960e537e03eee717feaac2774dd108a193e583ffc2cc7 WatchSource:0}: Error finding container 073d7b2a5107fe6ac95960e537e03eee717feaac2774dd108a193e583ffc2cc7: Status 404 returned error can't find the container with id 073d7b2a5107fe6ac95960e537e03eee717feaac2774dd108a193e583ffc2cc7
Oct 29 22:52:55.207348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.207305    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:55.207460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.207396    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:55.207460 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.207441    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:55.433396 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.433355    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" event={"ID":"2b6a29c6-375d-40d5-a2f2-144bc504e51f","Type":"ContainerStarted","Data":"67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5"}
Oct 29 22:52:55.433852 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.433425    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" event={"ID":"2b6a29c6-375d-40d5-a2f2-144bc504e51f","Type":"ContainerStarted","Data":"073d7b2a5107fe6ac95960e537e03eee717feaac2774dd108a193e583ffc2cc7"}
Oct 29 22:52:55.434762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.434732    4451 generic.go:356] "Generic (PLEG): container finished" podID="3a0a3325-9385-4384-b736-22139499892b" containerID="acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828" exitCode=137
Oct 29 22:52:55.434892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.434759    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:55.434892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.434817    4451 scope.go:122] "RemoveContainer" containerID="acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828"
Oct 29 22:52:55.446738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.446722    4451 scope.go:122] "RemoveContainer" containerID="acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828"
Oct 29 22:52:55.447039 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:55.447017    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828\": container with ID starting with acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828 not found: ID does not exist" containerID="acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828"
Oct 29 22:52:55.447166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.447050    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828"} err="failed to get container status \"acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828\": rpc error: code = NotFound desc = could not find container \"acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828\": container with ID starting with acf542ff27e1f3f47ebfdd6e10e0cd6cd90cd62d343be446555427dc97260828 not found: ID does not exist"
Oct 29 22:52:55.471303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.471224    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podStartSLOduration=1.47121213 podStartE2EDuration="1.47121213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:55.470097927 +0000 UTC m=+2315.413056633" watchObservedRunningTime="2026-10-29 22:52:55.47121213 +0000 UTC m=+2315.414170835"
Oct 29 22:52:55.481381 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:55.481356    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-d4316c\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="3a0a3325-9385-4384-b736-22139499892b" pod="opendatahub/test-curl-9092-d4316c"
Oct 29 22:52:55.881884 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:55.881851    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:52:56.167217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.167139    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0a3325-9385-4384-b736-22139499892b" path="/var/lib/kubelet/pods/3a0a3325-9385-4384-b736-22139499892b/volumes"
Oct 29 22:52:56.725498 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.725464    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:56.845570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.845534    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr\" (UniqueName: \"kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr\") pod \"50264019-541f-4668-b722-dda4db1ec046\" (UID: \"50264019-541f-4668-b722-dda4db1ec046\") "
Oct 29 22:52:56.847726 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.847697    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr" pod "50264019-541f-4668-b722-dda4db1ec046" (UID: "50264019-541f-4668-b722-dda4db1ec046"). InnerVolumeSpecName "kube-api-access-v8bpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:56.882727 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.882696    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:52:56.882863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.882784    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 29 22:52:56.947592 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:56.947518    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bpr\" (UniqueName: \"kubernetes.io/projected/50264019-541f-4668-b722-dda4db1ec046-kube-api-access-v8bpr\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:52:57.460136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.459539    4451 generic.go:356] "Generic (PLEG): container finished" podID="50264019-541f-4668-b722-dda4db1ec046" containerID="cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0" exitCode=137
Oct 29 22:52:57.460136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.459598    4451 scope.go:122] "RemoveContainer" containerID="cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0"
Oct 29 22:52:57.460136 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.459618    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:57.471785 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.471589    4451 scope.go:122] "RemoveContainer" containerID="cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0"
Oct 29 22:52:57.471902 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:57.471881    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0\": container with ID starting with cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0 not found: ID does not exist" containerID="cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0"
Oct 29 22:52:57.471980 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.471926    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0"} err="failed to get container status \"cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0\": rpc error: code = NotFound desc = could not find container \"cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0\": container with ID starting with cd0a08b48fd4b864a7663329d21808c02a8e9e8bfca816a21cad393cb60244f0 not found: ID does not exist"
Oct 29 22:52:57.481747 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:52:57.481717    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-72e0f7\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="50264019-541f-4668-b722-dda4db1ec046" pod="opendatahub/test-curl-9092-72e0f7"
Oct 29 22:52:57.881466 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.881435    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.881846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:57.881491    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 29 22:52:58.165609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:58.165529    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50264019-541f-4668-b722-dda4db1ec046" path="/var/lib/kubelet/pods/50264019-541f-4668-b722-dda4db1ec046/volumes"
Oct 29 22:52:58.882137 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:58.882099    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.882511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:58.882161    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 29 22:52:59.881558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:59.881525    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.881765 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:52:59.881583    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 29 22:53:00.881760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:00.881723    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.882186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:00.881780    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 29 22:53:01.881870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:01.881819    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.882286 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:01.881928    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 29 22:53:02.887254 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:02.887224    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:53:02.887696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:02.887499    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:53:02.888193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:02.888171    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 22:53:14.056294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.056252    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"]
Oct 29 22:53:14.060792 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.060698    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.089422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.089392    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"]
Oct 29 22:53:14.091036 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.091009    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-eb02df\"/\"default-dockercfg-khqv2\""
Oct 29 22:53:14.091162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.091081    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-eb02df\"/\"openshift-service-ca.crt\""
Oct 29 22:53:14.091162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.091112    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-eb02df\"/\"kube-root-ca.crt\""
Oct 29 22:53:14.091240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.091216    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-eb02df\"/\"auth-test-model-eb02df-kserve-self-signed-certs\""
Oct 29 22:53:14.153041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.153010    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"]
Oct 29 22:53:14.158486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.158461    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.185145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.177711    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3bd0f6-openshift-default-dockercfg-cmm6g\""
Oct 29 22:53:14.202310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.200813    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"]
Oct 29 22:53:14.222803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.222771    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.222989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.222850    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.223063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.223000    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.223106 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.223062    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.223169 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.223144    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.223307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.223290    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvzk\" (UniqueName: \"kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325092    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325220    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvzk\" (UniqueName: \"kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325266    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325303    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325393    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325507    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4msdz\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325545    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325587    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325644    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325709    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325753    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325821    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.339229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.325845    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.362932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.355184    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.362932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.355247    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.362932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.358549    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.362932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.358816    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.362932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.358969    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.362932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.359236    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.368491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.368460    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.371609 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.371586    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvzk\" (UniqueName: \"kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk\") pod \"auth-test-model-eb02df-kserve-6b5948d96f-6xnbs\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") " pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.393512 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.393214    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:14.456667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.456630    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.460567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.457538    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.460567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.457630    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.460567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.457632    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.460567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.458010    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.460567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.458116    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.461893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.457751    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.461893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.461051    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.461893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.461139    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.461893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.461202    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4msdz\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.461893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.461236    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.461893 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.461318    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.462298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.462040    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.462458 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.462433    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.466581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.466463    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.467702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.467654    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.479357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.479329    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.486712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.486659    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4msdz\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.535936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.521360    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:14.748624 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:53:14.748569    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3708043_58a0_470d_a435_6e8da5daae5d.slice/crio-fcacc04e4b525b28509e239b04858564822fc6927308884a44f192096bc99543 WatchSource:0}: Error finding container fcacc04e4b525b28509e239b04858564822fc6927308884a44f192096bc99543: Status 404 returned error can't find the container with id fcacc04e4b525b28509e239b04858564822fc6927308884a44f192096bc99543
Oct 29 22:53:14.752511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.752476    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"]
Oct 29 22:53:14.871749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.871724    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"]
Oct 29 22:53:14.873632 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:53:14.873593    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5130665c_f681_4734_9356_0046af52ee3c.slice/crio-a2f29e325113f9f60a4e4dcf6d66a526f08b24031ab73c7ad6722a66b01bfae3 WatchSource:0}: Error finding container a2f29e325113f9f60a4e4dcf6d66a526f08b24031ab73c7ad6722a66b01bfae3: Status 404 returned error can't find the container with id a2f29e325113f9f60a4e4dcf6d66a526f08b24031ab73c7ad6722a66b01bfae3
Oct 29 22:53:14.876788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.876755    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:14.876892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.876824    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:14.876892 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:14.876854    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:15.642529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:15.642488    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" event={"ID":"5130665c-f681-4734-9356-0046af52ee3c","Type":"ContainerStarted","Data":"03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5"}
Oct 29 22:53:15.643119 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:15.643092    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" event={"ID":"5130665c-f681-4734-9356-0046af52ee3c","Type":"ContainerStarted","Data":"a2f29e325113f9f60a4e4dcf6d66a526f08b24031ab73c7ad6722a66b01bfae3"}
Oct 29 22:53:15.646274 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:15.646254    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs" event={"ID":"b3708043-58a0-470d-a435-6e8da5daae5d","Type":"ContainerStarted","Data":"a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62"}
Oct 29 22:53:15.646427 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:15.646413    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs" event={"ID":"b3708043-58a0-470d-a435-6e8da5daae5d","Type":"ContainerStarted","Data":"fcacc04e4b525b28509e239b04858564822fc6927308884a44f192096bc99543"}
Oct 29 22:53:15.685450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:15.685402    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podStartSLOduration=1.685389585 podStartE2EDuration="1.685389585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:15.683809775 +0000 UTC m=+2335.626768481" watchObservedRunningTime="2026-10-29 22:53:15.685389585 +0000 UTC m=+2335.628348289"
Oct 29 22:53:15.723722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:15.723669    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs" podStartSLOduration=1.7236503029999999 podStartE2EDuration="1.723650303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:15.721482552 +0000 UTC m=+2335.664441280" watchObservedRunningTime="2026-10-29 22:53:15.723650303 +0000 UTC m=+2335.666609008"
Oct 29 22:53:16.399936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:16.393986    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:16.523415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:16.523380    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:16.525432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:16.525404    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 22:53:16.525570 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:16.525470    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 22:53:17.523529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:17.523474    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 22:53:17.524018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:17.523565    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 22:53:18.523240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:18.523202    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 22:53:18.523422 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:18.523264    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 22:53:18.758498 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:53:18.758462    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:53:19.524242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:19.524199    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 22:53:19.524880 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:19.524827    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 22:53:20.523454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:20.523419    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 22:53:20.524026 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:20.523473    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 22:53:21.679379 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:21.679341    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:21.680141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:21.680112    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:21.681277 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:21.681253    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 22:53:24.393904 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:24.393863    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:24.406721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:24.406694    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:24.631029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:24.630991    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"]
Oct 29 22:53:24.735655 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:24.735173    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:53:25.794220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:25.794184    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"]
Oct 29 22:53:33.832667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:33.832631    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"]
Oct 29 22:53:39.656776 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:39.656690    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" podUID="0254ffd1-ce60-471d-811e-3770cc3082b8" containerName="main" containerID="cri-o://d5e927dc19c7185a001fb1002dcc568e726dd848a8180692735a07aac64d0693" gracePeriod=45
Oct 29 22:53:39.836008 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:39.835968    4451 generic.go:356] "Generic (PLEG): container finished" podID="0254ffd1-ce60-471d-811e-3770cc3082b8" containerID="d5e927dc19c7185a001fb1002dcc568e726dd848a8180692735a07aac64d0693" exitCode=0
Oct 29 22:53:39.836205 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:39.836085    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" event={"ID":"0254ffd1-ce60-471d-811e-3770cc3082b8","Type":"ContainerDied","Data":"d5e927dc19c7185a001fb1002dcc568e726dd848a8180692735a07aac64d0693"}
Oct 29 22:53:40.112155 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.112120    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:53:40.239798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.239765    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs\") pod \"0254ffd1-ce60-471d-811e-3770cc3082b8\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") "
Oct 29 22:53:40.240022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.239897    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache\") pod \"0254ffd1-ce60-471d-811e-3770cc3082b8\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") "
Oct 29 22:53:40.240022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.239937    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm\") pod \"0254ffd1-ce60-471d-811e-3770cc3082b8\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") "
Oct 29 22:53:40.240022 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.239976    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s\" (UniqueName: \"kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s\") pod \"0254ffd1-ce60-471d-811e-3770cc3082b8\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") "
Oct 29 22:53:40.240199 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.240086    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home\") pod \"0254ffd1-ce60-471d-811e-3770cc3082b8\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") "
Oct 29 22:53:40.240260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.240230    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir\") pod \"0254ffd1-ce60-471d-811e-3770cc3082b8\" (UID: \"0254ffd1-ce60-471d-811e-3770cc3082b8\") "
Oct 29 22:53:40.240320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.240246    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache" pod "0254ffd1-ce60-471d-811e-3770cc3082b8" (UID: "0254ffd1-ce60-471d-811e-3770cc3082b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:40.240525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.240497    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home" pod "0254ffd1-ce60-471d-811e-3770cc3082b8" (UID: "0254ffd1-ce60-471d-811e-3770cc3082b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:40.240853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.240830    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir" pod "0254ffd1-ce60-471d-811e-3770cc3082b8" (UID: "0254ffd1-ce60-471d-811e-3770cc3082b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:40.241207 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.241180    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:40.241361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.241346    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:40.241453 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.241442    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:40.243818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.243768    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm" pod "0254ffd1-ce60-471d-811e-3770cc3082b8" (UID: "0254ffd1-ce60-471d-811e-3770cc3082b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:40.244242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.244217    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs" pod "0254ffd1-ce60-471d-811e-3770cc3082b8" (UID: "0254ffd1-ce60-471d-811e-3770cc3082b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:40.246398 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.246361    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s" pod "0254ffd1-ce60-471d-811e-3770cc3082b8" (UID: "0254ffd1-ce60-471d-811e-3770cc3082b8"). InnerVolumeSpecName "kube-api-access-8fz6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:40.343113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.343048    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0254ffd1-ce60-471d-811e-3770cc3082b8-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:40.343113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.343087    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fz6s\" (UniqueName: \"kubernetes.io/projected/0254ffd1-ce60-471d-811e-3770cc3082b8-kube-api-access-8fz6s\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:40.343113 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.343104    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0254ffd1-ce60-471d-811e-3770cc3082b8-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:40.826395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.826359    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" podUID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" containerName="main" containerID="cri-o://340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd" gracePeriod=45
Oct 29 22:53:40.846875 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.846617    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"
Oct 29 22:53:40.846875 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.846620    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" event={"ID":"0254ffd1-ce60-471d-811e-3770cc3082b8","Type":"ContainerDied","Data":"b1178ddb9e3debf148d00f3357db4644c5d3af29ba65c972643d9d7d137ac275"}
Oct 29 22:53:40.846875 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.846685    4451 scope.go:122] "RemoveContainer" containerID="d5e927dc19c7185a001fb1002dcc568e726dd848a8180692735a07aac64d0693"
Oct 29 22:53:40.921501 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.921456    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp" podUID="0254ffd1-ce60-471d-811e-3770cc3082b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:40.927276 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.927248    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"]
Oct 29 22:53:40.939762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:40.939728    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/auth-test-model-af260f-kserve-d854864c7-cd9sp"]
Oct 29 22:53:41.143660 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.143631    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:53:41.261219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261192    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir\") pod \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") "
Oct 29 22:53:41.261402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261229    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm\") pod \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") "
Oct 29 22:53:41.261402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261316    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs\") pod \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") "
Oct 29 22:53:41.261589 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261540    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home\") pod \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") "
Oct 29 22:53:41.261662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261596    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz\" (UniqueName: \"kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz\") pod \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") "
Oct 29 22:53:41.261662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261623    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir" pod "4f67cc43-b89b-4b47-9e9a-9e506dca8a23" (UID: "4f67cc43-b89b-4b47-9e9a-9e506dca8a23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:41.261786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261730    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache\") pod \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\" (UID: \"4f67cc43-b89b-4b47-9e9a-9e506dca8a23\") "
Oct 29 22:53:41.261863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.261807    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home" pod "4f67cc43-b89b-4b47-9e9a-9e506dca8a23" (UID: "4f67cc43-b89b-4b47-9e9a-9e506dca8a23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:41.262121 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.262092    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache" pod "4f67cc43-b89b-4b47-9e9a-9e506dca8a23" (UID: "4f67cc43-b89b-4b47-9e9a-9e506dca8a23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:41.262506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.262477    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:41.262616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.262510    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:41.262616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.262526    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:41.263856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.263833    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm" pod "4f67cc43-b89b-4b47-9e9a-9e506dca8a23" (UID: "4f67cc43-b89b-4b47-9e9a-9e506dca8a23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:41.263986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.263852    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs" pod "4f67cc43-b89b-4b47-9e9a-9e506dca8a23" (UID: "4f67cc43-b89b-4b47-9e9a-9e506dca8a23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:41.264157 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.264136    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz" pod "4f67cc43-b89b-4b47-9e9a-9e506dca8a23" (UID: "4f67cc43-b89b-4b47-9e9a-9e506dca8a23"). InnerVolumeSpecName "kube-api-access-rjjsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:41.363798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.363712    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjjsz\" (UniqueName: \"kubernetes.io/projected/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-kube-api-access-rjjsz\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:41.363798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.363743    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:41.363798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.363753    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f67cc43-b89b-4b47-9e9a-9e506dca8a23-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:41.858896 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.858854    4451 generic.go:356] "Generic (PLEG): container finished" podID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" containerID="340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd" exitCode=0
Oct 29 22:53:41.859371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.858902    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"
Oct 29 22:53:41.859371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.858938    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" event={"ID":"4f67cc43-b89b-4b47-9e9a-9e506dca8a23","Type":"ContainerDied","Data":"340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd"}
Oct 29 22:53:41.859371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.858980    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" event={"ID":"4f67cc43-b89b-4b47-9e9a-9e506dca8a23","Type":"ContainerDied","Data":"63fee78ef3f99971d566f22e14775901610c2d760e9c0ca40e5598c3069429f3"}
Oct 29 22:53:41.859371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.859002    4451 scope.go:122] "RemoveContainer" containerID="340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd"
Oct 29 22:53:41.876869 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.876847    4451 scope.go:122] "RemoveContainer" containerID="340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd"
Oct 29 22:53:41.877214 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:53:41.877191    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd\": container with ID starting with 340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd not found: ID does not exist" containerID="340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd"
Oct 29 22:53:41.877331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.877225    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd"} err="failed to get container status \"340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd\": rpc error: code = NotFound desc = could not find container \"340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd\": container with ID starting with 340378d0f2950a160725bbb740e58c28bdcaea8948dc6b50c9992cec977883fd not found: ID does not exist"
Oct 29 22:53:41.907526 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.907496    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"]
Oct 29 22:53:41.917407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:41.917381    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm"]
Oct 29 22:53:42.073346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:42.073306    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-9aafe6/rate-test-model-9aafe6-kserve-7bf5fbf957-txtjm" podUID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded"
Oct 29 22:53:42.164974 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:42.164868    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0254ffd1-ce60-471d-811e-3770cc3082b8" path="/var/lib/kubelet/pods/0254ffd1-ce60-471d-811e-3770cc3082b8/volumes"
Oct 29 22:53:42.165424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:42.165406    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" path="/var/lib/kubelet/pods/4f67cc43-b89b-4b47-9e9a-9e506dca8a23/volumes"
Oct 29 22:53:48.743818 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:53:48.743779    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:53:48.877779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:48.877715    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q" podUID="c12bca7d-9928-4dac-89ed-50c31ade29a5" containerName="main" containerID="cri-o://d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269" gracePeriod=45
Oct 29 22:53:49.161278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.161247    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:53:49.251826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.251744    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm\") pod \"c12bca7d-9928-4dac-89ed-50c31ade29a5\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") "
Oct 29 22:53:49.252065 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.251849    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache\") pod \"c12bca7d-9928-4dac-89ed-50c31ade29a5\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") "
Oct 29 22:53:49.252065 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.251893    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs\") pod \"c12bca7d-9928-4dac-89ed-50c31ade29a5\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") "
Oct 29 22:53:49.252065 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.251981    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir\") pod \"c12bca7d-9928-4dac-89ed-50c31ade29a5\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") "
Oct 29 22:53:49.252065 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.252052    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv\" (UniqueName: \"kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv\") pod \"c12bca7d-9928-4dac-89ed-50c31ade29a5\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") "
Oct 29 22:53:49.252345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.252110    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home\") pod \"c12bca7d-9928-4dac-89ed-50c31ade29a5\" (UID: \"c12bca7d-9928-4dac-89ed-50c31ade29a5\") "
Oct 29 22:53:49.253281 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.252952    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache" pod "c12bca7d-9928-4dac-89ed-50c31ade29a5" (UID: "c12bca7d-9928-4dac-89ed-50c31ade29a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:49.253281 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.253193    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir" pod "c12bca7d-9928-4dac-89ed-50c31ade29a5" (UID: "c12bca7d-9928-4dac-89ed-50c31ade29a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:49.253281 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.253257    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home" pod "c12bca7d-9928-4dac-89ed-50c31ade29a5" (UID: "c12bca7d-9928-4dac-89ed-50c31ade29a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:49.255527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.255502    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs" pod "c12bca7d-9928-4dac-89ed-50c31ade29a5" (UID: "c12bca7d-9928-4dac-89ed-50c31ade29a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:49.255628 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.255583    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm" pod "c12bca7d-9928-4dac-89ed-50c31ade29a5" (UID: "c12bca7d-9928-4dac-89ed-50c31ade29a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:49.257370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.257345    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv" pod "c12bca7d-9928-4dac-89ed-50c31ade29a5" (UID: "c12bca7d-9928-4dac-89ed-50c31ade29a5"). InnerVolumeSpecName "kube-api-access-4qjdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:49.354691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.354653    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:49.354691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.354685    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:49.354691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.354696    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12bca7d-9928-4dac-89ed-50c31ade29a5-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:49.354928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.354704    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:49.354928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.354713    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qjdv\" (UniqueName: \"kubernetes.io/projected/c12bca7d-9928-4dac-89ed-50c31ade29a5-kube-api-access-4qjdv\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:49.354928 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.354722    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12bca7d-9928-4dac-89ed-50c31ade29a5-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:53:49.934702 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.934670    4451 generic.go:356] "Generic (PLEG): container finished" podID="c12bca7d-9928-4dac-89ed-50c31ade29a5" containerID="d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269" exitCode=0
Oct 29 22:53:49.935170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.934719    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"
Oct 29 22:53:49.935170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.934758    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q" event={"ID":"c12bca7d-9928-4dac-89ed-50c31ade29a5","Type":"ContainerDied","Data":"d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269"}
Oct 29 22:53:49.935170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.934805    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q" event={"ID":"c12bca7d-9928-4dac-89ed-50c31ade29a5","Type":"ContainerDied","Data":"c7092a8398238bcee416a18c12e647f78cd267084eae8895ab540d4a19c341a7"}
Oct 29 22:53:49.935170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.934827    4451 scope.go:122] "RemoveContainer" containerID="d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269"
Oct 29 22:53:49.945798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.945778    4451 scope.go:122] "RemoveContainer" containerID="d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269"
Oct 29 22:53:49.946076 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:53:49.946056    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269\": container with ID starting with d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269 not found: ID does not exist" containerID="d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269"
Oct 29 22:53:49.946163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.946089    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269"} err="failed to get container status \"d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269\": rpc error: code = NotFound desc = could not find container \"d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269\": container with ID starting with d9a6c967206dfc5485c76d9fc127d174e81b1320a91ad8075bcac0cc2ad98269 not found: ID does not exist"
Oct 29 22:53:49.981494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.981461    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"]
Oct 29 22:53:49.991006 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:49.990976    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/auth-test-model-d68c2c-kserve-86575997b5-pz65q"]
Oct 29 22:53:50.166759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:50.166720    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12bca7d-9928-4dac-89ed-50c31ade29a5" path="/var/lib/kubelet/pods/c12bca7d-9928-4dac-89ed-50c31ade29a5/volumes"
Oct 29 22:53:55.685516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.685433    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"]
Oct 29 22:53:55.686415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686389    4451 cpu_manager.go:414] "containerMap: removing container" podUID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" containerName="main"
Oct 29 22:53:55.686415 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686414    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" containerName="main"
Oct 29 22:53:55.686577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686439    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c12bca7d-9928-4dac-89ed-50c31ade29a5" containerName="main"
Oct 29 22:53:55.686577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686444    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12bca7d-9928-4dac-89ed-50c31ade29a5" containerName="main"
Oct 29 22:53:55.686577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686454    4451 cpu_manager.go:414] "containerMap: removing container" podUID="0254ffd1-ce60-471d-811e-3770cc3082b8" containerName="main"
Oct 29 22:53:55.686577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686459    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0254ffd1-ce60-471d-811e-3770cc3082b8" containerName="main"
Oct 29 22:53:55.686577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686565    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f67cc43-b89b-4b47-9e9a-9e506dca8a23" containerName="main"
Oct 29 22:53:55.686577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686578    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12bca7d-9928-4dac-89ed-50c31ade29a5" containerName="main"
Oct 29 22:53:55.686800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.686584    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="0254ffd1-ce60-471d-811e-3770cc3082b8" containerName="main"
Oct 29 22:53:55.691885 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.691717    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.719934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.715346    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c87bf1\""
Oct 29 22:53:55.719934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.715507    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c87bf1-dockercfg-vn56m\""
Oct 29 22:53:55.719934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.718031    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"]
Oct 29 22:53:55.772024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.771967    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.772230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.772162    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74cf\" (UniqueName: \"kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.848461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.848427    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"]
Oct 29 22:53:55.852542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.852521    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:55.871846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.871822    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"]
Oct 29 22:53:55.873484 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.873457    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.873630 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.873612    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74cf\" (UniqueName: \"kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.874180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.874159    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.893449 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.893426    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74cf\" (UniqueName: \"kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:55.975085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.975013    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dq5q\" (UniqueName: \"kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q\") pod \"payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:55.975085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:55.975079    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:56.019116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.019090    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:56.077274 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.077240    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dq5q\" (UniqueName: \"kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q\") pod \"payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:56.077385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.077315    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:56.078339 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.078307    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:56.105334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.105271    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dq5q\" (UniqueName: \"kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q\") pod \"payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:56.167795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.167766    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:56.320025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.319996    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"]
Oct 29 22:53:56.320140 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:53:56.319999    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e48dfe_242c_4dc8_8992_398b634ed0d2.slice/crio-6fb5429defef30673827f6e80029ac95979a175460e55d1699a275aead592fbb WatchSource:0}: Error finding container 6fb5429defef30673827f6e80029ac95979a175460e55d1699a275aead592fbb: Status 404 returned error can't find the container with id 6fb5429defef30673827f6e80029ac95979a175460e55d1699a275aead592fbb
Oct 29 22:53:56.322181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.322163    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:53:56.472319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.472268    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"]
Oct 29 22:53:56.473110 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:53:56.473079    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aff0317_ebc2_46ad_9f61_5e2c550828a0.slice/crio-acc7ef9604a481d65d26f6a974defa49d53905ccfd19d1e883158d2e00c838f1 WatchSource:0}: Error finding container acc7ef9604a481d65d26f6a974defa49d53905ccfd19d1e883158d2e00c838f1: Status 404 returned error can't find the container with id acc7ef9604a481d65d26f6a974defa49d53905ccfd19d1e883158d2e00c838f1
Oct 29 22:53:56.984483 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.984380    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7" event={"ID":"b6e48dfe-242c-4dc8-8992-398b634ed0d2","Type":"ContainerStarted","Data":"6fb5429defef30673827f6e80029ac95979a175460e55d1699a275aead592fbb"}
Oct 29 22:53:56.986463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:56.986430    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456" event={"ID":"6aff0317-ebc2-46ad-9f61-5e2c550828a0","Type":"ContainerStarted","Data":"acc7ef9604a481d65d26f6a974defa49d53905ccfd19d1e883158d2e00c838f1"}
Oct 29 22:53:57.994412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:57.994085    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7" event={"ID":"b6e48dfe-242c-4dc8-8992-398b634ed0d2","Type":"ContainerStarted","Data":"c7c0082f58b9950e56559850b41058af8ea0fe1b7b41beccdfd8f6a8f241fe11"}
Oct 29 22:53:57.994412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:57.994169    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:53:57.995924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:57.995889    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456" event={"ID":"6aff0317-ebc2-46ad-9f61-5e2c550828a0","Type":"ContainerStarted","Data":"9cd3e0fa92aecbc6a71d919a8ae5eab1e181ba101ff47ac236b9350aff836056"}
Oct 29 22:53:57.996116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:57.996093    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:53:58.032330 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:58.032222    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7" podStartSLOduration=1.5971692480000002 podStartE2EDuration="3.032205034s" totalImagesPullingTime="1.435035786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:58.028082722 +0000 UTC m=+2377.971041434" watchObservedRunningTime="2026-10-29 22:53:58.032205034 +0000 UTC m=+2377.975163744"
Oct 29 22:53:58.064683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:53:58.064630    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456" podStartSLOduration=1.724542439 podStartE2EDuration="3.064613634s" totalImagesPullingTime="1.340071195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:58.061881694 +0000 UTC m=+2378.004840401" watchObservedRunningTime="2026-10-29 22:53:58.064613634 +0000 UTC m=+2378.007572342"
Oct 29 22:54:00.505886 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.505848    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"]
Oct 29 22:54:00.513107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.512219    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.532945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.532899    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"]
Oct 29 22:54:00.535474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.535444    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3ee017\"/\"auth-test-model-3ee017-kserve-self-signed-certs\""
Oct 29 22:54:00.535598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.535482    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3ee017\"/\"openshift-service-ca.crt\""
Oct 29 22:54:00.535598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.535508    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3ee017\"/\"kube-root-ca.crt\""
Oct 29 22:54:00.535598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.535572    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3ee017\"/\"default-dockercfg-5k97f\""
Oct 29 22:54:00.645072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.645035    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.645243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.645084    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.645307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.645240    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gxl\" (UniqueName: \"kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.645444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.645414    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.645511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.645499    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.645555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.645524    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.746904    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.747013    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.747041    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.747165    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.747211    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.747325    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gxl\" (UniqueName: \"kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.748183    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.748450    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.748810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.748666    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.764011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.763213    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.770941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.770569    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.778529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.776614    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gxl\" (UniqueName: \"kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl\") pod \"auth-test-model-3ee017-kserve-77865bd869-9gnx7\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") " pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.778529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.777856    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"]
Oct 29 22:54:00.827932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.823986    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"]
Oct 29 22:54:00.827932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.824110    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.874143 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.874103    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-9aafe6-openshift-default-dockercfg-6g2nq\""
Oct 29 22:54:00.874976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.874950    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:00.978188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.977887    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.978188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.977954    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.978188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.977990    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.978188 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.978021    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2wr\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.982266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.979258    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.982266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.979364    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.982266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.979423    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.982266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.979450    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:00.982266 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:00.979477    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081361    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081437    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081466    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081491    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081672    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081707    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081768 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081741    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081770    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2wr\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081817    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081829    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081873    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.081973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.081933    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.082255 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.082149    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.082575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.082551    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.084166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.084141    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.084261 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.084196    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.099210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.099185    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.099329 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.099285    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2wr\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.196091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.195824    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:01.209068 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:54:01.206564    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7af914f4_302d_4a76_9c8c_e68651e7c5d6.slice/crio-2b614c8cf29f569e29f90514ee4d48078fcb0e0d5956f4f2e1d199aabe5d4575 WatchSource:0}: Error finding container 2b614c8cf29f569e29f90514ee4d48078fcb0e0d5956f4f2e1d199aabe5d4575: Status 404 returned error can't find the container with id 2b614c8cf29f569e29f90514ee4d48078fcb0e0d5956f4f2e1d199aabe5d4575
Oct 29 22:54:01.209068 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.207263    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"]
Oct 29 22:54:01.526897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.526866    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"]
Oct 29 22:54:01.527876 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:54:01.527841    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e166b0_1cc0_44e8_821a_c1fddbf98fc6.slice/crio-a8deefd43d5dc5b7f751193987c31d862933ed866784cad92911f4b468130a13 WatchSource:0}: Error finding container a8deefd43d5dc5b7f751193987c31d862933ed866784cad92911f4b468130a13: Status 404 returned error can't find the container with id a8deefd43d5dc5b7f751193987c31d862933ed866784cad92911f4b468130a13
Oct 29 22:54:01.530778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.530745    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:54:01.530844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.530832    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:54:01.530887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:01.530863    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:54:02.038369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.038334    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7" event={"ID":"7af914f4-302d-4a76-9c8c-e68651e7c5d6","Type":"ContainerStarted","Data":"e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3"}
Oct 29 22:54:02.038544 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.038381    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7" event={"ID":"7af914f4-302d-4a76-9c8c-e68651e7c5d6","Type":"ContainerStarted","Data":"2b614c8cf29f569e29f90514ee4d48078fcb0e0d5956f4f2e1d199aabe5d4575"}
Oct 29 22:54:02.040010 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.039982    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" event={"ID":"00e166b0-1cc0-44e8-821a-c1fddbf98fc6","Type":"ContainerStarted","Data":"9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21"}
Oct 29 22:54:02.040118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.040035    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" event={"ID":"00e166b0-1cc0-44e8-821a-c1fddbf98fc6","Type":"ContainerStarted","Data":"a8deefd43d5dc5b7f751193987c31d862933ed866784cad92911f4b468130a13"}
Oct 29 22:54:02.081778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.081729    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7" podStartSLOduration=2.081712657 podStartE2EDuration="2.081712657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:02.079175844 +0000 UTC m=+2382.022134551" watchObservedRunningTime="2026-10-29 22:54:02.081712657 +0000 UTC m=+2382.024671375"
Oct 29 22:54:02.126941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.126878    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podStartSLOduration=2.126863335 podStartE2EDuration="2.126863335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:02.121276935 +0000 UTC m=+2382.064235637" watchObservedRunningTime="2026-10-29 22:54:02.126863335 +0000 UTC m=+2382.069822041"
Oct 29 22:54:02.200220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.199288    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:02.203103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.203061    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:02.203383 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.203141    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:02.876105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:02.876071    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:03.205657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:03.205114    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:03.205657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:03.205176    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:04.212937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:04.209960    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:04.212937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:04.210053    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:05.216050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:05.216007    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:05.216509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:05.216070    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:06.202938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.201998    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:06.202938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.202073    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:06.568075 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.567992    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"]
Oct 29 22:54:06.572849 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.572830    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.597677 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.597646    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"sub-test-model-af260f-kserve-self-signed-certs\""
Oct 29 22:54:06.597993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.597972    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"default-dockercfg-p84j2\""
Oct 29 22:54:06.598140 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.598025    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"openshift-service-ca.crt\""
Oct 29 22:54:06.598229 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.598108    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-af260f\"/\"kube-root-ca.crt\""
Oct 29 22:54:06.622031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.621991    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"]
Oct 29 22:54:06.676132 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.676092    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.676348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.676158    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.676431 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.676407    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.676480 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.676444    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtg5g\" (UniqueName: \"kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.676652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.676629    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.676704 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.676689    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778265    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778316    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778525    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778560    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtg5g\" (UniqueName: \"kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778668    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778719    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.778816 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.778785    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.779032 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.779004    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.779401 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.779375    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.780932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.780895    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.781414 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.781391    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.813259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.813225    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtg5g\" (UniqueName: \"kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g\") pod \"sub-test-model-af260f-kserve-648f545ccf-4svlw\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") " pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:06.898170 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:06.898132    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:07.196799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:07.196308    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:07.196799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:07.196369    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:07.329345 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:54:07.329306    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e971f33_1553_4029_962f_4db5627aad5b.slice/crio-10a382f1e336a38101c0f6d6a7503020f24aff2ad1925332f3bf4d6918676947 WatchSource:0}: Error finding container 10a382f1e336a38101c0f6d6a7503020f24aff2ad1925332f3bf4d6918676947: Status 404 returned error can't find the container with id 10a382f1e336a38101c0f6d6a7503020f24aff2ad1925332f3bf4d6918676947
Oct 29 22:54:07.330107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:07.330080    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"]
Oct 29 22:54:07.747738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:07.747649    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"]
Oct 29 22:54:08.087821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:08.087781    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw" event={"ID":"8e971f33-1553-4029-962f-4db5627aad5b","Type":"ContainerStarted","Data":"bb8247a2141f7fc8d2136bcf5f7e5670de599d52cbb40ee176f6c64d80a98584"}
Oct 29 22:54:08.087821 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:08.087823    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw" event={"ID":"8e971f33-1553-4029-962f-4db5627aad5b","Type":"ContainerStarted","Data":"10a382f1e336a38101c0f6d6a7503020f24aff2ad1925332f3bf4d6918676947"}
Oct 29 22:54:08.131111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:08.131053    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw" podStartSLOduration=2.131034683 podStartE2EDuration="2.131034683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:08.124717156 +0000 UTC m=+2388.067675874" watchObservedRunningTime="2026-10-29 22:54:08.131034683 +0000 UTC m=+2388.073993391"
Oct 29 22:54:08.196590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:08.196555    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:54:08.196744 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:08.196621    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 29 22:54:08.899567 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:08.899535    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:09.004790 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:09.004763    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:54:09.004992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:09.004829    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:54:09.213319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:09.213225    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:09.215035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:09.215008    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:09.215896 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:09.215874    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:54:10.757410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:10.757304    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"]
Oct 29 22:54:10.890127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:10.890093    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:10.946358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:10.946328    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:11.123882 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:11.123854    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:54:16.468190 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.468152    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9095-a86b40"]
Oct 29 22:54:16.474242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.474220    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:16.490907 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.490876    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9095-a86b40"]
Oct 29 22:54:16.496700 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.496675    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:54:16.548443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.548405    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhj8\" (UniqueName: \"kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8\") pod \"test-curl-9095-a86b40\" (UID: \"be6b232e-723b-4fc3-81c2-882b392684f8\") " pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:16.650371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.650330    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhj8\" (UniqueName: \"kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8\") pod \"test-curl-9095-a86b40\" (UID: \"be6b232e-723b-4fc3-81c2-882b392684f8\") " pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:16.668050 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.668013    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhj8\" (UniqueName: \"kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8\") pod \"test-curl-9095-a86b40\" (UID: \"be6b232e-723b-4fc3-81c2-882b392684f8\") " pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:16.789860 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.789786    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:16.900845 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.900815    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:16.916811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:16.916778    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:17.110106 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:17.110077    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9095-a86b40"]
Oct 29 22:54:17.197890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:17.197550    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9095-a86b40" event={"ID":"be6b232e-723b-4fc3-81c2-882b392684f8","Type":"ContainerStarted","Data":"54793e629f33289145adcb1fd1b11caac2ce669a0d442cbc70653fd1fd8a5c9d"}
Oct 29 22:54:17.212215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:17.212191    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:54:18.207491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:18.207453    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9095-a86b40" event={"ID":"be6b232e-723b-4fc3-81c2-882b392684f8","Type":"ContainerStarted","Data":"9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9"}
Oct 29 22:54:18.246617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:18.246564    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9095-a86b40" podStartSLOduration=2.246551465 podStartE2EDuration="2.246551465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:18.242747019 +0000 UTC m=+2398.185705721" watchObservedRunningTime="2026-10-29 22:54:18.246551465 +0000 UTC m=+2398.189510169"
Oct 29 22:54:18.747839 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:18.747798    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:54:19.077805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:19.077773    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9095-a86b40"]
Oct 29 22:54:19.083722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:19.083667    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9095-a86b40"]
Oct 29 22:54:20.221581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:20.221548    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9095-a86b40" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" containerName="test-curl-9095-a86b40" containerID="cri-o://9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9" gracePeriod=2
Oct 29 22:54:22.450448 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:22.450411    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:22.552154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:22.552074    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8\" (UniqueName: \"kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8\") pod \"be6b232e-723b-4fc3-81c2-882b392684f8\" (UID: \"be6b232e-723b-4fc3-81c2-882b392684f8\") "
Oct 29 22:54:22.554503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:22.554469    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8" pod "be6b232e-723b-4fc3-81c2-882b392684f8" (UID: "be6b232e-723b-4fc3-81c2-882b392684f8"). InnerVolumeSpecName "kube-api-access-8bhj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:22.654082 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:22.654052    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhj8\" (UniqueName: \"kubernetes.io/projected/be6b232e-723b-4fc3-81c2-882b392684f8-kube-api-access-8bhj8\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:22.777066 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:22.777017    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97" podUID="150ac800-4c96-4bb4-af90-c08ca6c25fbf" containerName="main" containerID="cri-o://884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1" gracePeriod=45
Oct 29 22:54:23.045179 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.045144    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:54:23.162529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162438    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache\") pod \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") "
Oct 29 22:54:23.162529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162486    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99\" (UniqueName: \"kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99\") pod \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") "
Oct 29 22:54:23.162735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162558    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm\") pod \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") "
Oct 29 22:54:23.162735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162645    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs\") pod \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") "
Oct 29 22:54:23.162735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162670    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir\") pod \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") "
Oct 29 22:54:23.162735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162703    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home\") pod \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\" (UID: \"150ac800-4c96-4bb4-af90-c08ca6c25fbf\") "
Oct 29 22:54:23.162735 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.162722    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache" pod "150ac800-4c96-4bb4-af90-c08ca6c25fbf" (UID: "150ac800-4c96-4bb4-af90-c08ca6c25fbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:23.163035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.163010    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir" pod "150ac800-4c96-4bb4-af90-c08ca6c25fbf" (UID: "150ac800-4c96-4bb4-af90-c08ca6c25fbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:23.163116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.163088    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home" pod "150ac800-4c96-4bb4-af90-c08ca6c25fbf" (UID: "150ac800-4c96-4bb4-af90-c08ca6c25fbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:23.163565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.163540    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.163697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.163570    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.163697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.163585    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.165011 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.164985    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm" pod "150ac800-4c96-4bb4-af90-c08ca6c25fbf" (UID: "150ac800-4c96-4bb4-af90-c08ca6c25fbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:23.165130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.165098    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs" pod "150ac800-4c96-4bb4-af90-c08ca6c25fbf" (UID: "150ac800-4c96-4bb4-af90-c08ca6c25fbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:23.165130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.165113    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99" pod "150ac800-4c96-4bb4-af90-c08ca6c25fbf" (UID: "150ac800-4c96-4bb4-af90-c08ca6c25fbf"). InnerVolumeSpecName "kube-api-access-zhr99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:23.242611 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.242578    4451 generic.go:356] "Generic (PLEG): container finished" podID="150ac800-4c96-4bb4-af90-c08ca6c25fbf" containerID="884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1" exitCode=0
Oct 29 22:54:23.242760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.242633    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"
Oct 29 22:54:23.242760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.242665    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97" event={"ID":"150ac800-4c96-4bb4-af90-c08ca6c25fbf","Type":"ContainerDied","Data":"884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1"}
Oct 29 22:54:23.242760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.242720    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97" event={"ID":"150ac800-4c96-4bb4-af90-c08ca6c25fbf","Type":"ContainerDied","Data":"b64c535a332fdd4e59c7c3d629b7b986fd512294cb6ebbfff66a3b48853d92aa"}
Oct 29 22:54:23.242760 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.242736    4451 scope.go:122] "RemoveContainer" containerID="884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1"
Oct 29 22:54:23.244601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.244394    4451 generic.go:356] "Generic (PLEG): container finished" podID="be6b232e-723b-4fc3-81c2-882b392684f8" containerID="9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9" exitCode=137
Oct 29 22:54:23.244601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.244442    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:23.257964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.257948    4451 scope.go:122] "RemoveContainer" containerID="884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1"
Oct 29 22:54:23.258274 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:23.258255    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1\": container with ID starting with 884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1 not found: ID does not exist" containerID="884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1"
Oct 29 22:54:23.258325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.258283    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1"} err="failed to get container status \"884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1\": rpc error: code = NotFound desc = could not find container \"884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1\": container with ID starting with 884a7ce60b1f6d61aff3451770cdb21274682b301ae7758c26d25b286d2b97b1 not found: ID does not exist"
Oct 29 22:54:23.258325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.258300    4451 scope.go:122] "RemoveContainer" containerID="9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9"
Oct 29 22:54:23.265870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.265841    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150ac800-4c96-4bb4-af90-c08ca6c25fbf-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.265992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.265875    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhr99\" (UniqueName: \"kubernetes.io/projected/150ac800-4c96-4bb4-af90-c08ca6c25fbf-kube-api-access-zhr99\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.265992 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.265889    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150ac800-4c96-4bb4-af90-c08ca6c25fbf-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.267802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.267788    4451 scope.go:122] "RemoveContainer" containerID="9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9"
Oct 29 22:54:23.268113 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:23.268090    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9\": container with ID starting with 9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9 not found: ID does not exist" containerID="9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9"
Oct 29 22:54:23.268192 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.268122    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9"} err="failed to get container status \"9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9\": rpc error: code = NotFound desc = could not find container \"9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9\": container with ID starting with 9a018c218f550110548ff1aa36e3b5b44c82495f9fb8d2fcef1d8f10423203e9 not found: ID does not exist"
Oct 29 22:54:23.268440 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:23.268420    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9095-a86b40\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:23.301040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.301014    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"]
Oct 29 22:54:23.305369 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:23.305343    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa3c6/sub-test-model-aaa3c6-kserve-97758679f-tvm97"]
Oct 29 22:54:23.312902 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:23.312874    4451 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9095-a86b40\" is forbidden: User \"system:node:ip-10-0-143-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-15.ec2.internal' and this object" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" pod="opendatahub/test-curl-9095-a86b40"
Oct 29 22:54:24.165402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:24.165368    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150ac800-4c96-4bb4-af90-c08ca6c25fbf" path="/var/lib/kubelet/pods/150ac800-4c96-4bb4-af90-c08ca6c25fbf/volumes"
Oct 29 22:54:24.165803 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:24.165782    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" path="/var/lib/kubelet/pods/be6b232e-723b-4fc3-81c2-882b392684f8/volumes"
Oct 29 22:54:25.781950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:25.781863    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd" podUID="be453da7-4c8a-4ba5-a09a-7362f7e0a499" containerName="main" containerID="cri-o://d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50" gracePeriod=45
Oct 29 22:54:26.078469 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.078445    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:54:26.127384 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.127352    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"]
Oct 29 22:54:26.128134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128110    4451 cpu_manager.go:414] "containerMap: removing container" podUID="be453da7-4c8a-4ba5-a09a-7362f7e0a499" containerName="main"
Oct 29 22:54:26.128134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128133    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be453da7-4c8a-4ba5-a09a-7362f7e0a499" containerName="main"
Oct 29 22:54:26.128319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128153    4451 cpu_manager.go:414] "containerMap: removing container" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" containerName="test-curl-9095-a86b40"
Oct 29 22:54:26.128319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128161    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" containerName="test-curl-9095-a86b40"
Oct 29 22:54:26.128319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128187    4451 cpu_manager.go:414] "containerMap: removing container" podUID="150ac800-4c96-4bb4-af90-c08ca6c25fbf" containerName="main"
Oct 29 22:54:26.128319 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128195    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150ac800-4c96-4bb4-af90-c08ca6c25fbf" containerName="main"
Oct 29 22:54:26.128464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128386    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="be453da7-4c8a-4ba5-a09a-7362f7e0a499" containerName="main"
Oct 29 22:54:26.128464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128400    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6b232e-723b-4fc3-81c2-882b392684f8" containerName="test-curl-9095-a86b40"
Oct 29 22:54:26.128464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.128408    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="150ac800-4c96-4bb4-af90-c08ca6c25fbf" containerName="main"
Oct 29 22:54:26.132162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.132142    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.146235 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.146206    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"]
Oct 29 22:54:26.150132 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.148378    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7be71d-serving-cert\""
Oct 29 22:54:26.209920 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.209876    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache\") pod \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") "
Oct 29 22:54:26.210127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.209951    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home\") pod \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") "
Oct 29 22:54:26.210127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.209980    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs\" (UniqueName: \"kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs\") pod \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") "
Oct 29 22:54:26.210127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210062    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm\") pod \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") "
Oct 29 22:54:26.210127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210101    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr\" (UniqueName: \"kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr\") pod \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") "
Oct 29 22:54:26.210352 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210178    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache" pod "be453da7-4c8a-4ba5-a09a-7362f7e0a499" (UID: "be453da7-4c8a-4ba5-a09a-7362f7e0a499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:26.210433 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210416    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir\") pod \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\" (UID: \"be453da7-4c8a-4ba5-a09a-7362f7e0a499\") "
Oct 29 22:54:26.210633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210614    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4b6l\" (UniqueName: \"kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.210705 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210681    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.210789 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210776    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.210891 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.210878    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.211372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.211197    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir" pod "be453da7-4c8a-4ba5-a09a-7362f7e0a499" (UID: "be453da7-4c8a-4ba5-a09a-7362f7e0a499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:26.211372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.211252    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.211372 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.211265    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home" pod "be453da7-4c8a-4ba5-a09a-7362f7e0a499" (UID: "be453da7-4c8a-4ba5-a09a-7362f7e0a499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:26.212474 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.212446    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs" pod "be453da7-4c8a-4ba5-a09a-7362f7e0a499" (UID: "be453da7-4c8a-4ba5-a09a-7362f7e0a499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:26.213116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.213091    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm" pod "be453da7-4c8a-4ba5-a09a-7362f7e0a499" (UID: "be453da7-4c8a-4ba5-a09a-7362f7e0a499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:26.213325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.213300    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr" pod "be453da7-4c8a-4ba5-a09a-7362f7e0a499" (UID: "be453da7-4c8a-4ba5-a09a-7362f7e0a499"). InnerVolumeSpecName "kube-api-access-xkwbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:26.272999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.272964    4451 generic.go:356] "Generic (PLEG): container finished" podID="be453da7-4c8a-4ba5-a09a-7362f7e0a499" containerID="d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50" exitCode=0
Oct 29 22:54:26.273160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.273034    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"
Oct 29 22:54:26.273160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.273042    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd" event={"ID":"be453da7-4c8a-4ba5-a09a-7362f7e0a499","Type":"ContainerDied","Data":"d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50"}
Oct 29 22:54:26.273160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.273095    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd" event={"ID":"be453da7-4c8a-4ba5-a09a-7362f7e0a499","Type":"ContainerDied","Data":"2f7193f53e27c6b2258bb4009d36575d94b9355d1bda69628d02008d56b802b8"}
Oct 29 22:54:26.273160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.273118    4451 scope.go:122] "RemoveContainer" containerID="d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50"
Oct 29 22:54:26.284231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.284215    4451 scope.go:122] "RemoveContainer" containerID="d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50"
Oct 29 22:54:26.284512 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:26.284494    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50\": container with ID starting with d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50 not found: ID does not exist" containerID="d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50"
Oct 29 22:54:26.284565 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.284520    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50"} err="failed to get container status \"d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50\": rpc error: code = NotFound desc = could not find container \"d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50\": container with ID starting with d7940aba5f2a300f9e2658163053f24dfdf39e1e32b5e23b8a4e6411b5cd1f50 not found: ID does not exist"
Oct 29 22:54:26.312658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.312588    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.312828 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.312800    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4b6l\" (UniqueName: \"kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.313013 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.312857    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.313092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313044    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.313157 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:26.313138    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-serving-cert: secret "maas-api-e2e-ipp-a-7be71d-serving-cert" not found
Oct 29 22:54:26.313214 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:26.313202    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls podName:1851e1e9-8320-40d3-92fe-52d8e8d62ef2 nodeName:}" failed. No retries permitted until 2026-10-29 22:54:26.813181704 +0000 UTC m=+2406.756140401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls") pod "maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" (UID: "1851e1e9-8320-40d3-92fe-52d8e8d62ef2") : secret "maas-api-e2e-ipp-a-7be71d-serving-cert" not found
Oct 29 22:54:26.313807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313238    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.313807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313258    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.313807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313270    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be453da7-4c8a-4ba5-a09a-7362f7e0a499-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.313807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313284    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be453da7-4c8a-4ba5-a09a-7362f7e0a499-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.313807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313296    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwbr\" (UniqueName: \"kubernetes.io/projected/be453da7-4c8a-4ba5-a09a-7362f7e0a499-kube-api-access-xkwbr\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.313807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.313703    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.317258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.316676    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.322863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.320513    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"]
Oct 29 22:54:26.325837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.325807    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3bd0f6/sub-test-model-3bd0f6-kserve-6c7d896b7f-wf4wd"]
Oct 29 22:54:26.341494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.341165    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4b6l\" (UniqueName: \"kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.821485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.821443    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:26.824060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:26.824035    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:27.058064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:27.058030    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:27.377258 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:27.377229    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"]
Oct 29 22:54:27.379873 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:54:27.379843    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1851e1e9_8320_40d3_92fe_52d8e8d62ef2.slice/crio-8be4c3353119a5d0628e9f8bce8101fbd564db83af1c24716a90db6371fec810 WatchSource:0}: Error finding container 8be4c3353119a5d0628e9f8bce8101fbd564db83af1c24716a90db6371fec810: Status 404 returned error can't find the container with id 8be4c3353119a5d0628e9f8bce8101fbd564db83af1c24716a90db6371fec810
Oct 29 22:54:28.167475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:28.167440    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be453da7-4c8a-4ba5-a09a-7362f7e0a499" path="/var/lib/kubelet/pods/be453da7-4c8a-4ba5-a09a-7362f7e0a499/volumes"
Oct 29 22:54:28.289208 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:28.289174    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" event={"ID":"1851e1e9-8320-40d3-92fe-52d8e8d62ef2","Type":"ContainerStarted","Data":"ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c"}
Oct 29 22:54:28.289405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:28.289225    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" event={"ID":"1851e1e9-8320-40d3-92fe-52d8e8d62ef2","Type":"ContainerStarted","Data":"8be4c3353119a5d0628e9f8bce8101fbd564db83af1c24716a90db6371fec810"}
Oct 29 22:54:28.289405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:28.289309    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:28.329741 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:28.329688    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" podStartSLOduration=1.9878641670000001 podStartE2EDuration="2.32967491s" totalImagesPullingTime="341.810743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:28.32582892 +0000 UTC m=+2408.268787638" watchObservedRunningTime="2026-10-29 22:54:28.32967491 +0000 UTC m=+2408.272633635"
Oct 29 22:54:34.302692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:34.302659    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 22:54:43.564338 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:43.564301    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"]
Oct 29 22:54:45.352872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:45.352832    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"]
Oct 29 22:54:48.743343 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:48.743295    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:54:58.597884 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.597838    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" podUID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" containerName="main" containerID="cri-o://f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b" gracePeriod=45
Oct 29 22:54:58.685374 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.685334    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" podUID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": dial tcp 10.133.0.118:8000: connect: connection refused"
Oct 29 22:54:58.919490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.919463    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:54:58.995434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.995400    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home\") pod \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") "
Oct 29 22:54:58.995631 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.995476    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs\") pod \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") "
Oct 29 22:54:58.995631 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.995539    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache\") pod \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") "
Oct 29 22:54:58.995631 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.995576    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm\") pod \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") "
Oct 29 22:54:58.995839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.995643    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir\") pod \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") "
Oct 29 22:54:58.995839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.995663    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr\" (UniqueName: \"kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr\") pod \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\" (UID: \"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1\") "
Oct 29 22:54:58.996621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.996255    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache" pod "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" (UID: "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:58.996621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.996412    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:58.997211 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.996866    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home" pod "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" (UID: "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:58.999325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.999128    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm" pod "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" (UID: "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:58.999325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.999168    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr" pod "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" (UID: "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1"). InnerVolumeSpecName "kube-api-access-g9jsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:58.999552 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:58.999524    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir" pod "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" (UID: "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:59.000344 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.000317    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs" pod "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" (UID: "6f94dd79-def6-4aca-b33d-6b4cdab3e9b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:59.098407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.098373    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.098407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.098402    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.098407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.098415    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.098692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.098427    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.098692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.098440    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jsr\" (UniqueName: \"kubernetes.io/projected/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1-kube-api-access-g9jsr\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:54:59.507334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.507290    4451 generic.go:356] "Generic (PLEG): container finished" podID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" containerID="f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b" exitCode=0
Oct 29 22:54:59.507334 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.507333    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" event={"ID":"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1","Type":"ContainerDied","Data":"f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b"}
Oct 29 22:54:59.507590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.507352    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"
Oct 29 22:54:59.507590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.507372    4451 scope.go:122] "RemoveContainer" containerID="f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b"
Oct 29 22:54:59.507590 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.507361    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx" event={"ID":"6f94dd79-def6-4aca-b33d-6b4cdab3e9b1","Type":"ContainerDied","Data":"eda39133c2099d909f745e457e0608775944e53fc9a0d79bb86c0be34837d9f8"}
Oct 29 22:54:59.518216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.518199    4451 scope.go:122] "RemoveContainer" containerID="f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b"
Oct 29 22:54:59.518496 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:59.518478    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b\": container with ID starting with f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b not found: ID does not exist" containerID="f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b"
Oct 29 22:54:59.518561 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.518506    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b"} err="failed to get container status \"f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b\": rpc error: code = NotFound desc = could not find container \"f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b\": container with ID starting with f96d51e27fe1042b20093e616cc6bb10bb687503715182770746e588ba5f150b not found: ID does not exist"
Oct 29 22:54:59.554526 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.554484    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"]
Oct 29 22:54:59.562969 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.562938    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/auth-test-model-f8ca96-kserve-7fc6567c6f-z6gxx"]
Oct 29 22:54:59.728147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.728114    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"]
Oct 29 22:54:59.729008 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.728968    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" containerName="main"
Oct 29 22:54:59.729008 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.728993    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" containerName="main"
Oct 29 22:54:59.729218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.729155    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" containerName="main"
Oct 29 22:54:59.734180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.734162    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.747149 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.747112    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a962d7-serving-cert\""
Oct 29 22:54:59.750380 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.750357    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"]
Oct 29 22:54:59.809385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.809303    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.809385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.809349    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.809580 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.809555    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjln\" (UniqueName: \"kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.809632 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.809589    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.911320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.911288    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.911320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.911327    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.911596 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:59.911440    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-serving-cert: secret "maas-api-e2e-ipp-b-a962d7-serving-cert" not found
Oct 29 22:54:59.911596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.911452    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjln\" (UniqueName: \"kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.911596 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:54:59.911503    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls podName:260a2e7f-c3a3-4735-9bdb-b5b79fa0de43 nodeName:}" failed. No retries permitted until 2026-10-29 22:55:00.411483976 +0000 UTC m=+2440.354442680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls") pod "maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" (UID: "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43") : secret "maas-api-e2e-ipp-b-a962d7-serving-cert" not found
Oct 29 22:54:59.911596 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.911521    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.912435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.912407    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.914779 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.914750    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:54:59.934633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:54:59.934598    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjln\" (UniqueName: \"kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:55:00.165959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.165902    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f94dd79-def6-4aca-b33d-6b4cdab3e9b1" path="/var/lib/kubelet/pods/6f94dd79-def6-4aca-b33d-6b4cdab3e9b1/volumes"
Oct 29 22:55:00.418259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.418175    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:55:00.420705 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.420674    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:55:00.446682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.446639    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6" podUID="6d404b47-2069-48ef-9a0d-1404f4feaff2" containerName="main" containerID="cri-o://5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de" gracePeriod=45
Oct 29 22:55:00.649622 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.649379    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:55:00.762367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.762339    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:55:00.826035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826003    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache\") pod \"6d404b47-2069-48ef-9a0d-1404f4feaff2\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") "
Oct 29 22:55:00.826314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826162    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir\") pod \"6d404b47-2069-48ef-9a0d-1404f4feaff2\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") "
Oct 29 22:55:00.826314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826247    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home\") pod \"6d404b47-2069-48ef-9a0d-1404f4feaff2\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") "
Oct 29 22:55:00.826314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826307    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs\") pod \"6d404b47-2069-48ef-9a0d-1404f4feaff2\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") "
Oct 29 22:55:00.826506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826348    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm\") pod \"6d404b47-2069-48ef-9a0d-1404f4feaff2\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") "
Oct 29 22:55:00.826506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826419    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz\" (UniqueName: \"kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz\") pod \"6d404b47-2069-48ef-9a0d-1404f4feaff2\" (UID: \"6d404b47-2069-48ef-9a0d-1404f4feaff2\") "
Oct 29 22:55:00.826506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.826483    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir" pod "6d404b47-2069-48ef-9a0d-1404f4feaff2" (UID: "6d404b47-2069-48ef-9a0d-1404f4feaff2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:00.827154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.827121    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:00.827293 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.827217    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home" pod "6d404b47-2069-48ef-9a0d-1404f4feaff2" (UID: "6d404b47-2069-48ef-9a0d-1404f4feaff2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:00.827366 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.827340    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache" pod "6d404b47-2069-48ef-9a0d-1404f4feaff2" (UID: "6d404b47-2069-48ef-9a0d-1404f4feaff2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:00.831107 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.831073    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm" pod "6d404b47-2069-48ef-9a0d-1404f4feaff2" (UID: "6d404b47-2069-48ef-9a0d-1404f4feaff2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:00.831243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.831131    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz" pod "6d404b47-2069-48ef-9a0d-1404f4feaff2" (UID: "6d404b47-2069-48ef-9a0d-1404f4feaff2"). InnerVolumeSpecName "kube-api-access-f4ksz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:00.834182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.834145    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs" pod "6d404b47-2069-48ef-9a0d-1404f4feaff2" (UID: "6d404b47-2069-48ef-9a0d-1404f4feaff2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:00.928927 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.928887    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:00.929063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.928936    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:00.929063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.928950    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d404b47-2069-48ef-9a0d-1404f4feaff2-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:00.929063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.928963    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d404b47-2069-48ef-9a0d-1404f4feaff2-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:00.929063 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:00.928976    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ksz\" (UniqueName: \"kubernetes.io/projected/6d404b47-2069-48ef-9a0d-1404f4feaff2-kube-api-access-f4ksz\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:01.004424 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:55:01.004391    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260a2e7f_c3a3_4735_9bdb_b5b79fa0de43.slice/crio-e82c3c30628e69b1c4dc059fab0a75148c2f99c97e6ed58ab5d76202c3869cc7 WatchSource:0}: Error finding container e82c3c30628e69b1c4dc059fab0a75148c2f99c97e6ed58ab5d76202c3869cc7: Status 404 returned error can't find the container with id e82c3c30628e69b1c4dc059fab0a75148c2f99c97e6ed58ab5d76202c3869cc7
Oct 29 22:55:01.004841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.004809    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"]
Oct 29 22:55:01.527205 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.527166    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" event={"ID":"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43","Type":"ContainerStarted","Data":"6e85c979e4925c173fca38e2ac383febf353c00fb6feeb18b48fca0b9134a898"}
Oct 29 22:55:01.527381 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.527271    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" event={"ID":"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43","Type":"ContainerStarted","Data":"e82c3c30628e69b1c4dc059fab0a75148c2f99c97e6ed58ab5d76202c3869cc7"}
Oct 29 22:55:01.527500 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.527468    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:55:01.528666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.528639    4451 generic.go:356] "Generic (PLEG): container finished" podID="6d404b47-2069-48ef-9a0d-1404f4feaff2" containerID="5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de" exitCode=0
Oct 29 22:55:01.528758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.528687    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"
Oct 29 22:55:01.528758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.528714    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6" event={"ID":"6d404b47-2069-48ef-9a0d-1404f4feaff2","Type":"ContainerDied","Data":"5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de"}
Oct 29 22:55:01.528758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.528750    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6" event={"ID":"6d404b47-2069-48ef-9a0d-1404f4feaff2","Type":"ContainerDied","Data":"4971c9aeaffd5b4d10329e48a12881dafa6fede4be0dc4317762ae30017884f7"}
Oct 29 22:55:01.528880 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.528767    4451 scope.go:122] "RemoveContainer" containerID="5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de"
Oct 29 22:55:01.541017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.540995    4451 scope.go:122] "RemoveContainer" containerID="5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de"
Oct 29 22:55:01.541282 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:55:01.541261    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de\": container with ID starting with 5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de not found: ID does not exist" containerID="5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de"
Oct 29 22:55:01.541341 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.541290    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de"} err="failed to get container status \"5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de\": rpc error: code = NotFound desc = could not find container \"5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de\": container with ID starting with 5227c08e1740d6341c5ec4ea82824e6b4a8bff17257002cd4cb37fdbca6480de not found: ID does not exist"
Oct 29 22:55:01.564675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.564634    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" podStartSLOduration=2.181581584 podStartE2EDuration="2.564622144s" totalImagesPullingTime="383.04056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:01.561044889 +0000 UTC m=+2441.504003594" watchObservedRunningTime="2026-10-29 22:55:01.564622144 +0000 UTC m=+2441.507580848"
Oct 29 22:55:01.593534 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.593509    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"]
Oct 29 22:55:01.599802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:01.599774    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/auth-test-model-c933cb-kserve-659668974f-jtwb6"]
Oct 29 22:55:02.166217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:02.166182    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d404b47-2069-48ef-9a0d-1404f4feaff2" path="/var/lib/kubelet/pods/6d404b47-2069-48ef-9a0d-1404f4feaff2/volumes"
Oct 29 22:55:07.562662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:07.562622    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:55:14.175952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.175900    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"]
Oct 29 22:55:14.176647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.176630    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6d404b47-2069-48ef-9a0d-1404f4feaff2" containerName="main"
Oct 29 22:55:14.176772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.176648    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d404b47-2069-48ef-9a0d-1404f4feaff2" containerName="main"
Oct 29 22:55:14.176834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.176781    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d404b47-2069-48ef-9a0d-1404f4feaff2" containerName="main"
Oct 29 22:55:14.183462 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.183426    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.202784 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.202754    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"]
Oct 29 22:55:14.205373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.204756    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"openshift-service-ca.crt\""
Oct 29 22:55:14.205373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.204878    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"default-dockercfg-bhj4t\""
Oct 29 22:55:14.205373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.205029    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"rate-test-model-f8ca96-kserve-self-signed-certs\""
Oct 29 22:55:14.205373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.205097    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f8ca96\"/\"kube-root-ca.crt\""
Oct 29 22:55:14.285033 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.284999    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.285237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.285166    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.285237 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.285212    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhmk\" (UniqueName: \"kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.285358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.285240    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.285436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.285418    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.285493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.285447    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387036    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387081    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387180    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387300    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387348    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhmk\" (UniqueName: \"kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387381    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387569    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387692 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387662    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.387750 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.387726    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.389943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.389885    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.390420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.390398    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.451990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.424660    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhmk\" (UniqueName: \"kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk\") pod \"rate-test-model-f8ca96-kserve-586cd65bd5-vctzg\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") " pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.512933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.508935    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:14.863951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:14.863903    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"]
Oct 29 22:55:14.866017 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:55:14.865989    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8be302e7_6277_4830_99a2_654c81400255.slice/crio-7cfe8e6b65c88818875611ff804e2964147dccb865aff6e7e7aeeee0ee38948a WatchSource:0}: Error finding container 7cfe8e6b65c88818875611ff804e2964147dccb865aff6e7e7aeeee0ee38948a: Status 404 returned error can't find the container with id 7cfe8e6b65c88818875611ff804e2964147dccb865aff6e7e7aeeee0ee38948a
Oct 29 22:55:15.421890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.421850    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"]
Oct 29 22:55:15.427114 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.427053    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.449018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.448983    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"]
Oct 29 22:55:15.455231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.455191    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"kube-root-ca.crt\""
Oct 29 22:55:15.455231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.455216    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"default-dockercfg-kgqsz\""
Oct 29 22:55:15.455231 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.455222    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"openshift-service-ca.crt\""
Oct 29 22:55:15.455443 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.455278    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d68c2c\"/\"sub-test-model-d68c2c-kserve-self-signed-certs\""
Oct 29 22:55:15.512134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.512087    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.512345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.512265    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.512437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.512405    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.512491 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.512445    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.512717 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.512691    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.512832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.512808    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndd7j\" (UniqueName: \"kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.614556 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.614516    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.614772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.614568    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.614844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.614771    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.614906 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.614867    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndd7j\" (UniqueName: \"kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.615025 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.614976    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.615085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.615027    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.615085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.615070    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.615245 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.615221    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.615464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.615435    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.617397 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.617375    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.618017 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.617997    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.632845 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.632809    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndd7j\" (UniqueName: \"kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j\") pod \"sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") " pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:15.662533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.662442    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg" event={"ID":"8be302e7-6277-4830-99a2-654c81400255","Type":"ContainerStarted","Data":"a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9"}
Oct 29 22:55:15.662533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.662500    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg" event={"ID":"8be302e7-6277-4830-99a2-654c81400255","Type":"ContainerStarted","Data":"7cfe8e6b65c88818875611ff804e2964147dccb865aff6e7e7aeeee0ee38948a"}
Oct 29 22:55:15.707996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.705749    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg" podStartSLOduration=1.705727618 podStartE2EDuration="1.705727618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:15.704347329 +0000 UTC m=+2455.647306035" watchObservedRunningTime="2026-10-29 22:55:15.705727618 +0000 UTC m=+2455.648686327"
Oct 29 22:55:15.745283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:15.745245    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:16.081701 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:16.081669    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"]
Oct 29 22:55:16.082089 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:55:16.082068    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae3d453c_44b8_49df_ac0a_3d8c539b20c3.slice/crio-0ec77347b89766e49a4f0a4eaf3ad303cbee640bd680707500dfdea4f2df3654 WatchSource:0}: Error finding container 0ec77347b89766e49a4f0a4eaf3ad303cbee640bd680707500dfdea4f2df3654: Status 404 returned error can't find the container with id 0ec77347b89766e49a4f0a4eaf3ad303cbee640bd680707500dfdea4f2df3654
Oct 29 22:55:16.510328 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:16.510246    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:16.673353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:16.673318    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" event={"ID":"ae3d453c-44b8-49df-ac0a-3d8c539b20c3","Type":"ContainerStarted","Data":"65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4"}
Oct 29 22:55:16.673353 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:16.673362    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" event={"ID":"ae3d453c-44b8-49df-ac0a-3d8c539b20c3","Type":"ContainerStarted","Data":"0ec77347b89766e49a4f0a4eaf3ad303cbee640bd680707500dfdea4f2df3654"}
Oct 29 22:55:16.708720 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:16.708669    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" podStartSLOduration=1.708655771 podStartE2EDuration="1.708655771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:16.707230893 +0000 UTC m=+2456.650189597" watchObservedRunningTime="2026-10-29 22:55:16.708655771 +0000 UTC m=+2456.651614476"
Oct 29 22:55:16.746215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:16.746186    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:18.743793 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:55:18.743754    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:55:24.510146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:24.509744    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:24.523493 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:24.523456    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:24.737496 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:24.737461    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:55:25.746542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:25.746511    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:25.759158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:25.759129    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:26.214395 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.214359    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"]
Oct 29 22:55:26.220300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.220268    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.246575 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.246546    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-920032-openshift-default-dockercfg-72mrh\""
Oct 29 22:55:26.252092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.252060    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"]
Oct 29 22:55:26.262306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262276    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262315    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262343    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6cw\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262371    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262464 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262397    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262686 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262463    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262774    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262835    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.262903 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.262884    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.363925 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.363874    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.363940    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.363969    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364120    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364159    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364192    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6cw\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364223    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364283 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364252    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364287    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364365    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364425    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364753 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364731    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.364807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.364773    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.365041 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.365023    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.366370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.366348    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.366878 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.366859    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.381301 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.381275    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.381421 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.381383    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6cw\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.535043 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.534955    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:26.750331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.750299    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:55:26.856954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.856906    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"]
Oct 29 22:55:26.857588 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:55:26.857558    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb53b1d8_2f2b_41c9_88f4_58ca148122be.slice/crio-82b7f25a82834d7f4dfd55c7b795ce0755bd6dc70dc56854eef39aa92a204af1 WatchSource:0}: Error finding container 82b7f25a82834d7f4dfd55c7b795ce0755bd6dc70dc56854eef39aa92a204af1: Status 404 returned error can't find the container with id 82b7f25a82834d7f4dfd55c7b795ce0755bd6dc70dc56854eef39aa92a204af1
Oct 29 22:55:26.860943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.860886    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:26.861020 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.860978    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:26.861058 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:26.861019    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:27.747376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:27.747329    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" event={"ID":"cb53b1d8-2f2b-41c9-88f4-58ca148122be","Type":"ContainerStarted","Data":"01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3"}
Oct 29 22:55:27.747574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:27.747418    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" event={"ID":"cb53b1d8-2f2b-41c9-88f4-58ca148122be","Type":"ContainerStarted","Data":"82b7f25a82834d7f4dfd55c7b795ce0755bd6dc70dc56854eef39aa92a204af1"}
Oct 29 22:55:27.819734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:27.819610    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podStartSLOduration=1.819575209 podStartE2EDuration="1.819575209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:27.819302787 +0000 UTC m=+2467.762261484" watchObservedRunningTime="2026-10-29 22:55:27.819575209 +0000 UTC m=+2467.762533915"
Oct 29 22:55:28.535199 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:28.535158    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:28.537230 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:28.537200    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:55:28.537400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:28.537248    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 29 22:55:29.536370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:29.535750    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:55:29.536370 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:29.535807    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 29 22:55:30.258604 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.258574    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"]
Oct 29 22:55:30.306054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.306020    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.329712 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.329677    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"]
Oct 29 22:55:30.337861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.337520    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"openshift-service-ca.crt\""
Oct 29 22:55:30.337861 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.337814    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"rate-test-model-c933cb-kserve-self-signed-certs\""
Oct 29 22:55:30.351028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.350250    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"default-dockercfg-88m6v\""
Oct 29 22:55:30.351028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.350337    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c933cb\"/\"kube-root-ca.crt\""
Oct 29 22:55:30.357329 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.357298    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.357618 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.357593    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.357804 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.357785    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.358055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.358034    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.358248 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.358225    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.358388 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.358369    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbbl\" (UniqueName: \"kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.461576 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.461535    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.461859 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.461837    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.462052 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.462032    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbbl\" (UniqueName: \"kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.462257 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.462238    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.462471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.462240    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.462697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.462676    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.462888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.462871    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.463219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.463180    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.463465 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.463418    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.466589 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.466336    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.473981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.473937    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.478874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.478834    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbbl\" (UniqueName: \"kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl\") pod \"rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") " pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:30.536697 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.536607    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:55:30.537345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.537310    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 29 22:55:30.652286 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:30.652239    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:31.172186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:31.171300    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"]
Oct 29 22:55:31.173587 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:55:31.173436    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6639a77b_8158_4f16_bb65_fc0aa80b259f.slice/crio-7e94de05ce4d92edb28778670b24ad8e96b5df1b344a6cb3321e9ff355ed8674 WatchSource:0}: Error finding container 7e94de05ce4d92edb28778670b24ad8e96b5df1b344a6cb3321e9ff355ed8674: Status 404 returned error can't find the container with id 7e94de05ce4d92edb28778670b24ad8e96b5df1b344a6cb3321e9ff355ed8674
Oct 29 22:55:31.536103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:31.536021    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:55:31.536103 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:31.536078    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 29 22:55:31.826680 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:31.826637    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv" event={"ID":"6639a77b-8158-4f16-bb65-fc0aa80b259f","Type":"ContainerStarted","Data":"630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750"}
Oct 29 22:55:31.827219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:31.826689    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv" event={"ID":"6639a77b-8158-4f16-bb65-fc0aa80b259f","Type":"ContainerStarted","Data":"7e94de05ce4d92edb28778670b24ad8e96b5df1b344a6cb3321e9ff355ed8674"}
Oct 29 22:55:31.864862 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:31.864810    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv" podStartSLOduration=1.8647929460000001 podStartE2EDuration="1.864792946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:31.86167492 +0000 UTC m=+2471.804633625" watchObservedRunningTime="2026-10-29 22:55:31.864792946 +0000 UTC m=+2471.807751653"
Oct 29 22:55:32.535528 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:32.535490    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:55:32.535721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:32.535552    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 29 22:55:32.653399 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:32.653359    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:32.665062 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:32.665027    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"]
Oct 29 22:55:32.665254 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:32.665230    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy" containerID="cri-o://00bd12c883a94f7a94429152b2d6032efeb9dbc6e4a6a43c3c5ad0dc01d8a6f6" gracePeriod=30
Oct 29 22:55:33.541240 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:33.541163    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:33.541635 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:33.541594    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:33.542441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:33.542421    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 22:55:33.992505 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:33.992467    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"]
Oct 29 22:55:33.997228 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:33.997203    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.010832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.010806    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-eb02df-openshift-default-dockercfg-8bq6z\""
Oct 29 22:55:34.029951 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.025802    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"]
Oct 29 22:55:34.171594 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171563    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.171766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171674    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.171766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171717    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2btl\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.171766 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171750    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.171897 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171778    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.171955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171944    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.171993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.171965    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.172039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.172002    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.172115 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.172089    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273206    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273626 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273595    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273649    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2btl\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273684    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273718    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273837    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.273950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273865    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274077 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.273982    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274077 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.274000    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.274075    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.274091    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.274195    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.274270    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.274894 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.274864    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.275772 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.275748    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.276318 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.276296    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.291557 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.291532    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.291696 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.291677    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2btl\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.312695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.312671    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:34.636158 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:55:34.636129    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e810219_babe_4d39_9b85_66d07fd14df6.slice/crio-b1490afa6ad682d68435c68e348c6c22515777277d19e9e0daffddc8881dec41 WatchSource:0}: Error finding container b1490afa6ad682d68435c68e348c6c22515777277d19e9e0daffddc8881dec41: Status 404 returned error can't find the container with id b1490afa6ad682d68435c68e348c6c22515777277d19e9e0daffddc8881dec41
Oct 29 22:55:34.637238 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.637208    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"]
Oct 29 22:55:34.639150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.639119    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:34.639223 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.639193    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:34.639268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.639221    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:34.863940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.863056    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" event={"ID":"9e810219-babe-4d39-9b85-66d07fd14df6","Type":"ContainerStarted","Data":"f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec"}
Oct 29 22:55:34.863940 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.863117    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" event={"ID":"9e810219-babe-4d39-9b85-66d07fd14df6","Type":"ContainerStarted","Data":"b1490afa6ad682d68435c68e348c6c22515777277d19e9e0daffddc8881dec41"}
Oct 29 22:55:34.905650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:34.905534    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podStartSLOduration=1.905514784 podStartE2EDuration="1.905514784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:34.902312441 +0000 UTC m=+2474.845271160" watchObservedRunningTime="2026-10-29 22:55:34.905514784 +0000 UTC m=+2474.848473491"
Oct 29 22:55:35.315300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:35.315185    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:35.317555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:35.317518    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:55:35.317740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:35.317590    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 22:55:36.315442 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:36.315381    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:55:36.316016 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:36.315456    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 22:55:37.317318 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:37.317276    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:55:37.317757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:37.317355    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 22:55:37.885447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:37.885418    4451 generic.go:356] "Generic (PLEG): container finished" podID="885ace3b-2569-476a-96b2-a17db6689b6b" containerID="00bd12c883a94f7a94429152b2d6032efeb9dbc6e4a6a43c3c5ad0dc01d8a6f6" exitCode=0
Oct 29 22:55:37.885616 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:37.885495    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" event={"ID":"885ace3b-2569-476a-96b2-a17db6689b6b","Type":"ContainerDied","Data":"00bd12c883a94f7a94429152b2d6032efeb9dbc6e4a6a43c3c5ad0dc01d8a6f6"}
Oct 29 22:55:38.052573 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.052541    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:55:38.153998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.153969    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154005    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154055    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154093    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154126    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154456    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154506    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:38.154631 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154600    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154698    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:38.154988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154721    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:38.154988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154745    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154775    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:38.154988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154806    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy\") pod \"885ace3b-2569-476a-96b2-a17db6689b6b\" (UID: \"885ace3b-2569-476a-96b2-a17db6689b6b\") "
Oct 29 22:55:38.154988 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.154958    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:38.157034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.155433    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.157034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.155462    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.157034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.155475    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.157034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.155484    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.157034 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.155492    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/885ace3b-2569-476a-96b2-a17db6689b6b-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.157272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.157088    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:55:38.157272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.157108    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:38.157640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.157621    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "kube-api-access-rvkd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:38.158593 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.158570    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token" pod "885ace3b-2569-476a-96b2-a17db6689b6b" (UID: "885ace3b-2569-476a-96b2-a17db6689b6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:38.257191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.257147    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.257191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.257183    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/885ace3b-2569-476a-96b2-a17db6689b6b-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.257191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.257199    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvkd8\" (UniqueName: \"kubernetes.io/projected/885ace3b-2569-476a-96b2-a17db6689b6b-kube-api-access-rvkd8\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.257444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.257213    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/885ace3b-2569-476a-96b2-a17db6689b6b-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:55:38.313719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.313247    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:55:38.313719 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.313317    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 22:55:38.909148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.909115    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"
Oct 29 22:55:38.909764 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.909734    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9" event={"ID":"885ace3b-2569-476a-96b2-a17db6689b6b","Type":"ContainerDied","Data":"3202a8912e2a0ef1b2fe2d031766b8bad1a4d4267303633849163537d700caeb"}
Oct 29 22:55:38.909933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.909899    4451 scope.go:122] "RemoveContainer" containerID="00bd12c883a94f7a94429152b2d6032efeb9dbc6e4a6a43c3c5ad0dc01d8a6f6"
Oct 29 22:55:38.957216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.957187    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"]
Oct 29 22:55:38.967161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:38.967137    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hrms9"]
Oct 29 22:55:39.315373 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:39.315294    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:55:39.315715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:39.315657    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 22:55:40.166032 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:40.165999    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" path="/var/lib/kubelet/pods/885ace3b-2569-476a-96b2-a17db6689b6b/volumes"
Oct 29 22:55:40.315292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:40.315251    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:55:40.315497 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:40.315328    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 22:55:40.653138 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:40.653093    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:40.665564 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:40.665539    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:40.941749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:40.941680    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:55:41.318930 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:41.318844    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:41.319338 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:41.319318    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:41.320035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:55:41.320014    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 22:55:48.746718 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:55:48.746677    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:56:10.035508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.035473    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"]
Oct 29 22:56:10.036166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.036151    4451 cpu_manager.go:414] "containerMap: removing container" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy"
Oct 29 22:56:10.036166 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.036166    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy"
Oct 29 22:56:10.036310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.036300    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="885ace3b-2569-476a-96b2-a17db6689b6b" containerName="istio-proxy"
Oct 29 22:56:10.041531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.041511    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.066350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.066304    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"]
Oct 29 22:56:10.111331 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111293    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.111523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111339    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.111577 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111524    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.111639 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111575    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.111763 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111740    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.111839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111804    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrdd\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.111890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.111843    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.112068 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.112008    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.112241 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.112071    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213405    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213516    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213568    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrdd\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213608    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213674    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213757 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213727    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213810    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.213863 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.213835    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.214028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.214007    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.214156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.214128    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.214343 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.214315    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.214451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.214401    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.214451 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.214432    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.214582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.214564    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.216024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.216003    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.216106 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.216080    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.230876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.230852    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.231023 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.231003    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrdd\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.375178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.375141    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:10.694348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.694317    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"]
Oct 29 22:56:10.694621 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:56:10.694598    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a8d410_df76_4882_9764_3227df225944.slice/crio-85f2822b2316d1c2edf17a59d48635818ac6e6f96085bf1342fbfe3b53486db2 WatchSource:0}: Error finding container 85f2822b2316d1c2edf17a59d48635818ac6e6f96085bf1342fbfe3b53486db2: Status 404 returned error can't find the container with id 85f2822b2316d1c2edf17a59d48635818ac6e6f96085bf1342fbfe3b53486db2
Oct 29 22:56:10.697225 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.697194    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:10.697300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.697256    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:10.697300 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:10.697282    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:11.199536 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.199474    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" event={"ID":"e9a8d410-df76-4882-9764-3227df225944","Type":"ContainerStarted","Data":"c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d"}
Oct 29 22:56:11.200078 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.199562    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" event={"ID":"e9a8d410-df76-4882-9764-3227df225944","Type":"ContainerStarted","Data":"85f2822b2316d1c2edf17a59d48635818ac6e6f96085bf1342fbfe3b53486db2"}
Oct 29 22:56:11.244723 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.244669    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podStartSLOduration=1.244654908 podStartE2EDuration="1.244654908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:11.242310169 +0000 UTC m=+2511.185268884" watchObservedRunningTime="2026-10-29 22:56:11.244654908 +0000 UTC m=+2511.187613612"
Oct 29 22:56:11.375470 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.375307    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:11.377299 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.377271    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:11.377441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.377325    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:11.638624 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.638584    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"]
Oct 29 22:56:11.644511 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.644487    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.660572 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.660522    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d68c2c-openshift-default-dockercfg-28wt6\""
Oct 29 22:56:11.672585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.672560    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"]
Oct 29 22:56:11.737605 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737561    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.737810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737625    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.737810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737655    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.737810 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737722    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.738000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737836    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.738000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737895    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkt2n\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.738000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737948    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.738000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.737996    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.738213 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.738141    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.839887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.839851    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.839949    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkt2n\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.839980    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840014    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840119    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840221    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840257    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840297    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840419 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840373    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840728 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840705    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.840811 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.840748    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.841104 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.841080    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.841158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.841101    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.841210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.841193    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.842529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.842466    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.842983 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.842965    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.857939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.857884    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkt2n\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.858031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.858003    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:11.960722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:11.960638    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:12.326116 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:56:12.326077    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392db7df_d6ab_4577_93fe_358d4f076821.slice/crio-9acaa696414273443438dfb5f4cb75ad892ad4dfe3b09515b9c5a7ab2b7ca4c1 WatchSource:0}: Error finding container 9acaa696414273443438dfb5f4cb75ad892ad4dfe3b09515b9c5a7ab2b7ca4c1: Status 404 returned error can't find the container with id 9acaa696414273443438dfb5f4cb75ad892ad4dfe3b09515b9c5a7ab2b7ca4c1
Oct 29 22:56:12.326560 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:12.326229    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"]
Oct 29 22:56:12.329518 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:12.329482    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:12.329633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:12.329550    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:12.329633 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:12.329580    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:12.376204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:12.376177    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:12.376316 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:12.376247    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:13.239647 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.239613    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" event={"ID":"392db7df-d6ab-4577-93fe-358d4f076821","Type":"ContainerStarted","Data":"0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259"}
Oct 29 22:56:13.239853 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.239672    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" event={"ID":"392db7df-d6ab-4577-93fe-358d4f076821","Type":"ContainerStarted","Data":"9acaa696414273443438dfb5f4cb75ad892ad4dfe3b09515b9c5a7ab2b7ca4c1"}
Oct 29 22:56:13.327935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.327547    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podStartSLOduration=2.327523557 podStartE2EDuration="2.327523557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:13.288846972 +0000 UTC m=+2513.231805682" watchObservedRunningTime="2026-10-29 22:56:13.327523557 +0000 UTC m=+2513.270482265"
Oct 29 22:56:13.378683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.378643    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.378879 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.378718    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:13.961040 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.961002    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:13.963033 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.963003    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.963184 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:13.963066    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 29 22:56:14.376243 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:14.376195    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.376705 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:14.376264    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:14.961559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:14.961523    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.961749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:14.961585    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 29 22:56:15.375620 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:15.375580    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.375834 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:15.375638    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:15.961986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:15.961944    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.962392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:15.961999    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 29 22:56:16.375775 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:16.375736    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:16.375986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:16.375794    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:16.961055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:16.961013    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:56:16.961239 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:16.961068    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 29 22:56:17.376086 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:17.376046    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:56:17.376547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:17.376109    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 29 22:56:17.961472 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:17.961436    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:56:17.961653 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:17.961494    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 29 22:56:18.381347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:18.381316    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:18.381954 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:18.381787    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:18.382598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:18.382580    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 22:56:18.748194 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:56:18.748107    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:56:18.961959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:18.961928    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:56:18.962138 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:18.961979    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 29 22:56:19.970530 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:19.970496    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:19.972722 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:19.972692    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:19.974105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:19.974078    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 22:56:38.402537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.402501    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"]
Oct 29 22:56:38.408888 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.408867    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.432260 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.432230    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4796a9-serving-cert\""
Oct 29 22:56:38.437966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.437937    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"]
Oct 29 22:56:38.505437 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.505404    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrvm\" (UniqueName: \"kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.505617 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.505561    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.505671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.505649    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.505791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.505776    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.614102 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.614050    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.615485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.615438    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.615675 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.615554    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrvm\" (UniqueName: \"kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.615749 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.615704    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.615986 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:56:38.615956    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-serving-cert: secret "maas-api-e2e-ipp-b-4796a9-serving-cert" not found
Oct 29 22:56:38.616133 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:56:38.616045    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls podName:b62cafb1-4b88-4125-91ae-72b574f847f3 nodeName:}" failed. No retries permitted until 2026-10-29 22:56:39.116022166 +0000 UTC m=+2539.058980851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls") pod "maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" (UID: "b62cafb1-4b88-4125-91ae-72b574f847f3") : secret "maas-api-e2e-ipp-b-4796a9-serving-cert" not found
Oct 29 22:56:38.617026 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.616997    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.621161 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.621133    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:38.635814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:38.635784    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrvm\" (UniqueName: \"kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:39.124721 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:39.124688    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:39.128949 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:39.128901    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:39.323802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:39.323761    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:39.655349 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:39.648871    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"]
Oct 29 22:56:40.454691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:40.454651    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" event={"ID":"b62cafb1-4b88-4125-91ae-72b574f847f3","Type":"ContainerStarted","Data":"358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7"}
Oct 29 22:56:40.454872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:40.454708    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" event={"ID":"b62cafb1-4b88-4125-91ae-72b574f847f3","Type":"ContainerStarted","Data":"0d7533f520062462b25d759a023d91b3e9827317117d6e1d60eac5362c9318f9"}
Oct 29 22:56:40.454872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:40.454759    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:40.493325 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:40.493277    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" podStartSLOduration=2.168995219 podStartE2EDuration="2.493263153s" totalImagesPullingTime="324.267934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:40.489197225 +0000 UTC m=+2540.432155929" watchObservedRunningTime="2026-10-29 22:56:40.493263153 +0000 UTC m=+2540.436221858"
Oct 29 22:56:46.468180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:46.468144    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:56:48.744144 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:56:48.744102    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:56:55.409148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.409111    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"]
Oct 29 22:56:55.414118 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.414096    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.433998 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.433967    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"]
Oct 29 22:56:55.438959 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.438934    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3ee017-openshift-default-dockercfg-xz768\""
Oct 29 22:56:55.558295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558253    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558321    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558452    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558508 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558498    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558517    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558656    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558731 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558699    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6rl\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558735    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.558778 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.558750    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660346    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660395    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660464    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660500    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660526    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660580    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660643 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660607    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6rl\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660658    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660683    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.660968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.660878    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.661100 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.661057    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.661139 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.661114    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.661174 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.661149    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.661215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.661191    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.663023 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.663000    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.663364 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.663344    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.677281 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.677252    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.677641 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.677623    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6rl\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:55.730085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:55.730036    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:56.052652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.052621    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"]
Oct 29 22:56:56.052775 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:56:56.052675    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7598412d_8b77_48fb_8d20_ee60f736228e.slice/crio-bb8a20fc7c048b906957baf955532e3c5532ce7a4eac19a987d83c26f5ec265b WatchSource:0}: Error finding container bb8a20fc7c048b906957baf955532e3c5532ce7a4eac19a987d83c26f5ec265b: Status 404 returned error can't find the container with id bb8a20fc7c048b906957baf955532e3c5532ce7a4eac19a987d83c26f5ec265b
Oct 29 22:56:56.055762 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.055731    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:56.055846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.055796    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:56.055846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.055825    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:56.584410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.584376    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" event={"ID":"7598412d-8b77-48fb-8d20-ee60f736228e","Type":"ContainerStarted","Data":"02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2"}
Oct 29 22:56:56.584837 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.584431    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" event={"ID":"7598412d-8b77-48fb-8d20-ee60f736228e","Type":"ContainerStarted","Data":"bb8a20fc7c048b906957baf955532e3c5532ce7a4eac19a987d83c26f5ec265b"}
Oct 29 22:56:56.643461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.643404    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podStartSLOduration=1.6433903239999998 podStartE2EDuration="1.643390324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:56.632183789 +0000 UTC m=+2556.575142494" watchObservedRunningTime="2026-10-29 22:56:56.643390324 +0000 UTC m=+2556.586349087"
Oct 29 22:56:56.731060 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:56.731032    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:56:57.732660 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:57.732628    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.733160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:57.732682    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 29 22:56:58.730390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:58.730354    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.730597 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:58.730409    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 29 22:56:59.730335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:59.730300    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.730754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:56:59.730354    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 29 22:57:00.730362 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:00.730323    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:57:00.730872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:00.730384    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 29 22:57:01.748835 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:01.748795    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:57:01.749533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:01.749495    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 29 22:57:02.731215 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:02.731178    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:57:02.731436 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:02.731240    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 29 22:57:03.736571 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:03.736539    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:57:03.737032 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:03.737017    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:57:03.737733 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:03.737714    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 22:57:11.358952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.358899    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"]
Oct 29 22:57:11.364614 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.364589    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.386203 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.386179    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-85862e-serving-cert\""
Oct 29 22:57:11.386393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.386363    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"]
Oct 29 22:57:11.463812 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.463774    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.463994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.463950    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdn52\" (UniqueName: \"kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.464178 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.464153    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.464233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.464207    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.568156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.567752    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.568156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.567945    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdn52\" (UniqueName: \"kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.568459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.568166    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.568459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.568213    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.568597 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:11.568554    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-serving-cert: secret "maas-api-e2e-ipp-b-85862e-serving-cert" not found
Oct 29 22:57:11.568655 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:11.568626    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls podName:41d9ebb5-d732-4eb5-86a7-f088a59bfd04 nodeName:}" failed. No retries permitted until 2026-10-29 22:57:12.068608437 +0000 UTC m=+2572.011567128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls") pod "maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" (UID: "41d9ebb5-d732-4eb5-86a7-f088a59bfd04") : secret "maas-api-e2e-ipp-b-85862e-serving-cert" not found
Oct 29 22:57:11.568761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.568738    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.571193 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.571167    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:11.585802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:11.585753    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdn52\" (UniqueName: \"kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:12.075723 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.075682    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:12.078220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.078197    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:12.144233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.144196    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"]
Oct 29 22:57:12.162975 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.162943    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.181730 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.181683    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.182088 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182065    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.182265 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182249    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.182381 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182366    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.182510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182493    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.182635 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182620    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.182758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182743    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.183009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.182990    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.183308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.183288    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwld\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.197021 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.194848    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-02720a-openshift-default-dockercfg-gzwqr\""
Oct 29 22:57:12.226049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.226012    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"]
Oct 29 22:57:12.284623 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.284592    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:12.285226 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285200    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285372    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwld\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285459 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285443    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285562 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285523    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285630 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285579    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285630 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285612    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285657    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285742 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285699    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.285851 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.285747    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.286165 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.286134    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.286666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.286358    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.286666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.286581    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.286666 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.286627    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.286858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.286834    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.288089 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.288067    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.288905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.288878    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.303310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.303284    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwld\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.303747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.303726    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.508829 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.508792    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:12.638204 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.638152    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"]
Oct 29 22:57:12.718662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.718622    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" event={"ID":"41d9ebb5-d732-4eb5-86a7-f088a59bfd04","Type":"ContainerStarted","Data":"5ced344b595f8f03fc0fa8beca78dcbdae9d0b2aba69032d10e991f8dfc6ccdc"}
Oct 29 22:57:12.892668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.892643    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"]
Oct 29 22:57:12.894458 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:57:12.894426    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc316150f_802e_4b9e_97eb_2611bfdf2ae3.slice/crio-a741b8a34ff1ef43be72c9000abe8d91bd4bc121f09e0e3fbc6c7cff6ecb2ef8 WatchSource:0}: Error finding container a741b8a34ff1ef43be72c9000abe8d91bd4bc121f09e0e3fbc6c7cff6ecb2ef8: Status 404 returned error can't find the container with id a741b8a34ff1ef43be72c9000abe8d91bd4bc121f09e0e3fbc6c7cff6ecb2ef8
Oct 29 22:57:12.899069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.899030    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:12.899173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.899115    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:12.899173 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:12.899165    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:13.728094 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.728052    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" event={"ID":"c316150f-802e-4b9e-97eb-2611bfdf2ae3","Type":"ContainerStarted","Data":"638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b"}
Oct 29 22:57:13.728612 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.728119    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" event={"ID":"c316150f-802e-4b9e-97eb-2611bfdf2ae3","Type":"ContainerStarted","Data":"a741b8a34ff1ef43be72c9000abe8d91bd4bc121f09e0e3fbc6c7cff6ecb2ef8"}
Oct 29 22:57:13.729966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.729938    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" event={"ID":"41d9ebb5-d732-4eb5-86a7-f088a59bfd04","Type":"ContainerStarted","Data":"d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e"}
Oct 29 22:57:13.730105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.730071    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:13.784802 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.784701    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podStartSLOduration=1.7846832510000001 podStartE2EDuration="1.784683251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:13.774119227 +0000 UTC m=+2573.717077967" watchObservedRunningTime="2026-10-29 22:57:13.784683251 +0000 UTC m=+2573.727641956"
Oct 29 22:57:13.871943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.864888    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" podStartSLOduration=2.343163129 podStartE2EDuration="2.864866314s" totalImagesPullingTime="521.703185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:13.827008244 +0000 UTC m=+2573.769966951" watchObservedRunningTime="2026-10-29 22:57:13.864866314 +0000 UTC m=+2573.807825020"
Oct 29 22:57:13.886994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:13.876715    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"]
Oct 29 22:57:14.514504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:14.514239    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:14.521091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:14.519002    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.521091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:14.519056    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 29 22:57:15.513934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:15.511294    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 29 22:57:15.513934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:15.511359    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 29 22:57:16.514967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:16.514158    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 29 22:57:16.514967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:16.514223    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 29 22:57:17.509936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:17.509237    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 29 22:57:17.509936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:17.509296    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 29 22:57:18.514935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:18.511209    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 29 22:57:18.514935 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:18.511276    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 29 22:57:18.752139 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:18.752069    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:57:19.519156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:19.519117    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:19.520844 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:19.520821    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:19.522348 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:19.522328    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 22:57:19.747558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:19.747531    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 22:57:29.095144 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.095093    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs" podUID="b3708043-58a0-470d-a435-6e8da5daae5d" containerName="main" containerID="cri-o://a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62" gracePeriod=45
Oct 29 22:57:29.430685 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.430657    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:57:29.535475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.535441    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs\") pod \"b3708043-58a0-470d-a435-6e8da5daae5d\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") "
Oct 29 22:57:29.535662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.535608    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir\") pod \"b3708043-58a0-470d-a435-6e8da5daae5d\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") "
Oct 29 22:57:29.535662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.535653    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk\" (UniqueName: \"kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk\") pod \"b3708043-58a0-470d-a435-6e8da5daae5d\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") "
Oct 29 22:57:29.535788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.535688    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache\") pod \"b3708043-58a0-470d-a435-6e8da5daae5d\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") "
Oct 29 22:57:29.535788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.535728    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home\") pod \"b3708043-58a0-470d-a435-6e8da5daae5d\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") "
Oct 29 22:57:29.535887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.535851    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir" pod "b3708043-58a0-470d-a435-6e8da5daae5d" (UID: "b3708043-58a0-470d-a435-6e8da5daae5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:29.536080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.536056    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm\") pod \"b3708043-58a0-470d-a435-6e8da5daae5d\" (UID: \"b3708043-58a0-470d-a435-6e8da5daae5d\") "
Oct 29 22:57:29.536454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.536153    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home" pod "b3708043-58a0-470d-a435-6e8da5daae5d" (UID: "b3708043-58a0-470d-a435-6e8da5daae5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:29.536454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.536206    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache" pod "b3708043-58a0-470d-a435-6e8da5daae5d" (UID: "b3708043-58a0-470d-a435-6e8da5daae5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:29.537545 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.537517    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:29.537657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.537547    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:29.537657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.537561    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:29.537991 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.537967    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs" pod "b3708043-58a0-470d-a435-6e8da5daae5d" (UID: "b3708043-58a0-470d-a435-6e8da5daae5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:29.538200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.538181    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk" pod "b3708043-58a0-470d-a435-6e8da5daae5d" (UID: "b3708043-58a0-470d-a435-6e8da5daae5d"). InnerVolumeSpecName "kube-api-access-jjvzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:29.538547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.538530    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm" pod "b3708043-58a0-470d-a435-6e8da5daae5d" (UID: "b3708043-58a0-470d-a435-6e8da5daae5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:29.639059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.638997    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3708043-58a0-470d-a435-6e8da5daae5d-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:29.639059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.639024    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3708043-58a0-470d-a435-6e8da5daae5d-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:29.639059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.639036    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvzk\" (UniqueName: \"kubernetes.io/projected/b3708043-58a0-470d-a435-6e8da5daae5d-kube-api-access-jjvzk\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:29.847024 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.846987    4451 generic.go:356] "Generic (PLEG): container finished" podID="b3708043-58a0-470d-a435-6e8da5daae5d" containerID="a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62" exitCode=0
Oct 29 22:57:29.847186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.847034    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"
Oct 29 22:57:29.847186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.847068    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs" event={"ID":"b3708043-58a0-470d-a435-6e8da5daae5d","Type":"ContainerDied","Data":"a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62"}
Oct 29 22:57:29.847186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.847116    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs" event={"ID":"b3708043-58a0-470d-a435-6e8da5daae5d","Type":"ContainerDied","Data":"fcacc04e4b525b28509e239b04858564822fc6927308884a44f192096bc99543"}
Oct 29 22:57:29.847186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.847137    4451 scope.go:122] "RemoveContainer" containerID="a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62"
Oct 29 22:57:29.858461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.858443    4451 scope.go:122] "RemoveContainer" containerID="a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62"
Oct 29 22:57:29.858752 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:29.858731    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62\": container with ID starting with a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62 not found: ID does not exist" containerID="a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62"
Oct 29 22:57:29.858846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.858758    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62"} err="failed to get container status \"a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62\": rpc error: code = NotFound desc = could not find container \"a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62\": container with ID starting with a95eb22d5319bc983c9bad503b1f326d166d5383e3450da31cc48ff5f73d4a62 not found: ID does not exist"
Oct 29 22:57:29.894232 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.894173    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"]
Oct 29 22:57:29.903738 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:29.903711    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-eb02df/auth-test-model-eb02df-kserve-6b5948d96f-6xnbs"]
Oct 29 22:57:30.167473 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:30.167397    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3708043-58a0-470d-a435-6e8da5daae5d" path="/var/lib/kubelet/pods/b3708043-58a0-470d-a435-6e8da5daae5d/volumes"
Oct 29 22:57:31.104652 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:31.104600    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"]
Oct 29 22:57:36.204890 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.204858    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"]
Oct 29 22:57:36.206531 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.206507    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b3708043-58a0-470d-a435-6e8da5daae5d" containerName="main"
Oct 29 22:57:36.206657 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.206532    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3708043-58a0-470d-a435-6e8da5daae5d" containerName="main"
Oct 29 22:57:36.206715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.206692    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3708043-58a0-470d-a435-6e8da5daae5d" containerName="main"
Oct 29 22:57:36.211826 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.211801    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.233943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.233869    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b8f54c-serving-cert\""
Oct 29 22:57:36.240671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.240641    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"]
Oct 29 22:57:36.265244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.265193    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqmg\" (UniqueName: \"kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.265454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.265360    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.265816 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.265774    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.265962 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.265845    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.367163 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.367123    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.367345 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.367209    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.367490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.367473    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqmg\" (UniqueName: \"kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.367583 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.367565    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.367842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.367818    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.369780 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.369754    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.369952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.369936    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.385233 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.385211    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqmg\" (UniqueName: \"kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg\") pod \"maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.528876 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.528761    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:36.844598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.844564    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"]
Oct 29 22:57:36.902090 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:36.902005    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw" event={"ID":"fcad3545-c689-4336-8fba-719bdd911ae3","Type":"ContainerStarted","Data":"53a75e819bb4ff390dd750d7c7bbcc9079ae733d8847d85c52c4b260e1df66b7"}
Oct 29 22:57:37.910242 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:37.910201    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw" event={"ID":"fcad3545-c689-4336-8fba-719bdd911ae3","Type":"ContainerStarted","Data":"0b25017c09ebdb5992ccd874d289cd0fbcecbc0d23d85ed82b1671932ba32078"}
Oct 29 22:57:37.910634 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:37.910272    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:37.956867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:37.956805    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw" podStartSLOduration=1.553659605 podStartE2EDuration="1.956785736s" totalImagesPullingTime="403.126131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:37.951353422 +0000 UTC m=+2597.894312128" watchObservedRunningTime="2026-10-29 22:57:37.956785736 +0000 UTC m=+2597.899744449"
Oct 29 22:57:43.957350 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:43.957318    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 22:57:46.129046 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.128988    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7" podUID="7af914f4-302d-4a76-9c8c-e68651e7c5d6" containerName="main" containerID="cri-o://e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3" gracePeriod=45
Oct 29 22:57:46.428848 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.428821    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:57:46.556199 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556170    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home\") pod \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") "
Oct 29 22:57:46.556382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556301    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs\") pod \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") "
Oct 29 22:57:46.556382 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556373    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache\") pod \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") "
Oct 29 22:57:46.556489 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556447    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home" pod "7af914f4-302d-4a76-9c8c-e68651e7c5d6" (UID: "7af914f4-302d-4a76-9c8c-e68651e7c5d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:46.556560 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556547    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl\" (UniqueName: \"kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl\") pod \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") "
Oct 29 22:57:46.556640 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556571    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir\") pod \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") "
Oct 29 22:57:46.556689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556641    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm\") pod \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\" (UID: \"7af914f4-302d-4a76-9c8c-e68651e7c5d6\") "
Oct 29 22:57:46.556689 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556652    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache" pod "7af914f4-302d-4a76-9c8c-e68651e7c5d6" (UID: "7af914f4-302d-4a76-9c8c-e68651e7c5d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:46.556866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.556845    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir" pod "7af914f4-302d-4a76-9c8c-e68651e7c5d6" (UID: "7af914f4-302d-4a76-9c8c-e68651e7c5d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:46.557292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.557270    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:46.557292 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.557293    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:46.557448 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.557306    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:46.558767 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.558741    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs" pod "7af914f4-302d-4a76-9c8c-e68651e7c5d6" (UID: "7af914f4-302d-4a76-9c8c-e68651e7c5d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:46.558866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.558820    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm" pod "7af914f4-302d-4a76-9c8c-e68651e7c5d6" (UID: "7af914f4-302d-4a76-9c8c-e68651e7c5d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:57:46.558866 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.558829    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl" pod "7af914f4-302d-4a76-9c8c-e68651e7c5d6" (UID: "7af914f4-302d-4a76-9c8c-e68651e7c5d6"). InnerVolumeSpecName "kube-api-access-52gxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:46.658905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.658825    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52gxl\" (UniqueName: \"kubernetes.io/projected/7af914f4-302d-4a76-9c8c-e68651e7c5d6-kube-api-access-52gxl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:46.658905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.658859    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af914f4-302d-4a76-9c8c-e68651e7c5d6-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:46.658905 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:46.658869    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af914f4-302d-4a76-9c8c-e68651e7c5d6-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:47.026308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.026226    4451 generic.go:356] "Generic (PLEG): container finished" podID="7af914f4-302d-4a76-9c8c-e68651e7c5d6" containerID="e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3" exitCode=0
Oct 29 22:57:47.026308 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.026272    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"
Oct 29 22:57:47.026547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.026318    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7" event={"ID":"7af914f4-302d-4a76-9c8c-e68651e7c5d6","Type":"ContainerDied","Data":"e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3"}
Oct 29 22:57:47.026547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.026362    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7" event={"ID":"7af914f4-302d-4a76-9c8c-e68651e7c5d6","Type":"ContainerDied","Data":"2b614c8cf29f569e29f90514ee4d48078fcb0e0d5956f4f2e1d199aabe5d4575"}
Oct 29 22:57:47.026547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.026386    4451 scope.go:122] "RemoveContainer" containerID="e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3"
Oct 29 22:57:47.037433 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.037416    4451 scope.go:122] "RemoveContainer" containerID="e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3"
Oct 29 22:57:47.037709 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:47.037688    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3\": container with ID starting with e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3 not found: ID does not exist" containerID="e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3"
Oct 29 22:57:47.037805 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.037714    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3"} err="failed to get container status \"e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3\": rpc error: code = NotFound desc = could not find container \"e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3\": container with ID starting with e8aba71c30f72250bb527a480757343be4fd2330272a596ab8a71585ee831af3 not found: ID does not exist"
Oct 29 22:57:47.072870 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.072835    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"]
Oct 29 22:57:47.082147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:47.082123    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3ee017/auth-test-model-3ee017-kserve-77865bd869-9gnx7"]
Oct 29 22:57:48.165120 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:48.165090    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af914f4-302d-4a76-9c8c-e68651e7c5d6" path="/var/lib/kubelet/pods/7af914f4-302d-4a76-9c8c-e68651e7c5d6/volumes"
Oct 29 22:57:48.746228 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:48.746191    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:57:50.517080 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:50.517040    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"]
Oct 29 22:57:50.517807 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:50.517753    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9" podUID="3494f4d4-87e6-4d73-88b2-2a32d6db7554" containerName="maas-api" containerID="cri-o://a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed" gracePeriod=30
Oct 29 22:57:50.674924 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:50.674883    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"]
Oct 29 22:57:50.675611 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:50.675372    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456" podUID="6aff0317-ebc2-46ad-9f61-5e2c550828a0" containerName="payload-processing" containerID="cri-o://9cd3e0fa92aecbc6a71d919a8ae5eab1e181ba101ff47ac236b9350aff836056" gracePeriod=30
Oct 29 22:57:50.709989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:50.699494    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"]
Oct 29 22:57:50.709989 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:50.699717    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7" podUID="b6e48dfe-242c-4dc8-8992-398b634ed0d2" containerName="payload-pre-processing" containerID="cri-o://c7c0082f58b9950e56559850b41058af8ea0fe1b7b41beccdfd8f6a8f241fe11" gracePeriod=30
Oct 29 22:57:51.249938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.248812    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:57:51.250145 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.250058    4451 generic.go:356] "Generic (PLEG): container finished" podID="b6e48dfe-242c-4dc8-8992-398b634ed0d2" containerID="c7c0082f58b9950e56559850b41058af8ea0fe1b7b41beccdfd8f6a8f241fe11" exitCode=0
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.250413    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7" event={"ID":"b6e48dfe-242c-4dc8-8992-398b634ed0d2","Type":"ContainerDied","Data":"c7c0082f58b9950e56559850b41058af8ea0fe1b7b41beccdfd8f6a8f241fe11"}
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.252645    4451 generic.go:356] "Generic (PLEG): container finished" podID="3494f4d4-87e6-4d73-88b2-2a32d6db7554" containerID="a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed" exitCode=0
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.252717    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9" event={"ID":"3494f4d4-87e6-4d73-88b2-2a32d6db7554","Type":"ContainerDied","Data":"a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed"}
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.252757    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9" event={"ID":"3494f4d4-87e6-4d73-88b2-2a32d6db7554","Type":"ContainerDied","Data":"928377aaf81b74e69d7e2244fbad1f4a8b1804c9302cca5b35bed4d3b95024be"}
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.252788    4451 scope.go:122] "RemoveContainer" containerID="a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed"
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.252999    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.256451    4451 generic.go:356] "Generic (PLEG): container finished" podID="6aff0317-ebc2-46ad-9f61-5e2c550828a0" containerID="9cd3e0fa92aecbc6a71d919a8ae5eab1e181ba101ff47ac236b9350aff836056" exitCode=0
Oct 29 22:57:51.259963 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.256489    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456" event={"ID":"6aff0317-ebc2-46ad-9f61-5e2c550828a0","Type":"ContainerDied","Data":"9cd3e0fa92aecbc6a71d919a8ae5eab1e181ba101ff47ac236b9350aff836056"}
Oct 29 22:57:51.309054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.306353    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz\" (UniqueName: \"kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz\") pod \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") "
Oct 29 22:57:51.309054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.306587    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca\") pod \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") "
Oct 29 22:57:51.309054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.306628    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs\") pod \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") "
Oct 29 22:57:51.309054 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.306741    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls\") pod \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\" (UID: \"3494f4d4-87e6-4d73-88b2-2a32d6db7554\") "
Oct 29 22:57:51.310774 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.310651    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca" pod "3494f4d4-87e6-4d73-88b2-2a32d6db7554" (UID: "3494f4d4-87e6-4d73-88b2-2a32d6db7554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:51.324441 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.324386    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls" pod "3494f4d4-87e6-4d73-88b2-2a32d6db7554" (UID: "3494f4d4-87e6-4d73-88b2-2a32d6db7554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:51.325324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.325299    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3494f4d4-87e6-4d73-88b2-2a32d6db7554-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.325435 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.325326    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.343073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.343024    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs" pod "3494f4d4-87e6-4d73-88b2-2a32d6db7554" (UID: "3494f4d4-87e6-4d73-88b2-2a32d6db7554"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:51.349359 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.348746    4451 scope.go:122] "RemoveContainer" containerID="a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed"
Oct 29 22:57:51.349990 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:57:51.349559    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed\": container with ID starting with a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed not found: ID does not exist" containerID="a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed"
Oct 29 22:57:51.349990 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.349605    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed"} err="failed to get container status \"a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed\": rpc error: code = NotFound desc = could not find container \"a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed\": container with ID starting with a22cd6dfe53d1fac5a402a8c467edd53a8f2b3be92c88752d832a41ebe7940ed not found: ID does not exist"
Oct 29 22:57:51.351134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.351089    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz" pod "3494f4d4-87e6-4d73-88b2-2a32d6db7554" (UID: "3494f4d4-87e6-4d73-88b2-2a32d6db7554"). InnerVolumeSpecName "kube-api-access-2qdlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:51.384037 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.383315    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:57:51.387507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.387485    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.426546    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf\" (UniqueName: \"kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf\") pod \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") "
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.426689    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume\") pod \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\" (UID: \"b6e48dfe-242c-4dc8-8992-398b634ed0d2\") "
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.426879    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q\" (UniqueName: \"kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q\") pod \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") "
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.427103    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume\") pod \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\" (UID: \"6aff0317-ebc2-46ad-9f61-5e2c550828a0\") "
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.427959    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3494f4d4-87e6-4d73-88b2-2a32d6db7554-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.428042    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdlz\" (UniqueName: \"kubernetes.io/projected/3494f4d4-87e6-4d73-88b2-2a32d6db7554-kube-api-access-2qdlz\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.429175 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.428670    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume" pod "b6e48dfe-242c-4dc8-8992-398b634ed0d2" (UID: "b6e48dfe-242c-4dc8-8992-398b634ed0d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:51.429976 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.429806    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume" pod "6aff0317-ebc2-46ad-9f61-5e2c550828a0" (UID: "6aff0317-ebc2-46ad-9f61-5e2c550828a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:51.435055 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.433275    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q" pod "6aff0317-ebc2-46ad-9f61-5e2c550828a0" (UID: "6aff0317-ebc2-46ad-9f61-5e2c550828a0"). InnerVolumeSpecName "kube-api-access-4dq5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:51.438244 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.438211    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf" pod "b6e48dfe-242c-4dc8-8992-398b634ed0d2" (UID: "b6e48dfe-242c-4dc8-8992-398b634ed0d2"). InnerVolumeSpecName "kube-api-access-v74cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:51.530085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.530006    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dq5q\" (UniqueName: \"kubernetes.io/projected/6aff0317-ebc2-46ad-9f61-5e2c550828a0-kube-api-access-4dq5q\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.530085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.530036    4451 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aff0317-ebc2-46ad-9f61-5e2c550828a0-plugins-config-volume\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.530085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.530046    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v74cf\" (UniqueName: \"kubernetes.io/projected/b6e48dfe-242c-4dc8-8992-398b634ed0d2-kube-api-access-v74cf\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.530085 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.530055    4451 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e48dfe-242c-4dc8-8992-398b634ed0d2-plugins-config-volume\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.598455 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.598416    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"]
Oct 29 22:57:51.602272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:51.602247    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87bf1-676854cc8-fv2r9"]
Oct 29 22:57:52.166210 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.166179    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3494f4d4-87e6-4d73-88b2-2a32d6db7554" path="/var/lib/kubelet/pods/3494f4d4-87e6-4d73-88b2-2a32d6db7554/volumes"
Oct 29 22:57:52.265126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.265098    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"
Oct 29 22:57:52.265271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.265137    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7" event={"ID":"b6e48dfe-242c-4dc8-8992-398b634ed0d2","Type":"ContainerDied","Data":"6fb5429defef30673827f6e80029ac95979a175460e55d1699a275aead592fbb"}
Oct 29 22:57:52.265271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.265197    4451 scope.go:122] "RemoveContainer" containerID="c7c0082f58b9950e56559850b41058af8ea0fe1b7b41beccdfd8f6a8f241fe11"
Oct 29 22:57:52.268185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.268159    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456" event={"ID":"6aff0317-ebc2-46ad-9f61-5e2c550828a0","Type":"ContainerDied","Data":"acc7ef9604a481d65d26f6a974defa49d53905ccfd19d1e883158d2e00c838f1"}
Oct 29 22:57:52.268343 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.268205    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"
Oct 29 22:57:52.276250 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.276087    4451 scope.go:122] "RemoveContainer" containerID="9cd3e0fa92aecbc6a71d919a8ae5eab1e181ba101ff47ac236b9350aff836056"
Oct 29 22:57:52.321146 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.321121    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"]
Oct 29 22:57:52.330277 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.330255    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-6bc965df8c-79jh7"]
Oct 29 22:57:52.355521 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.355480    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"]
Oct 29 22:57:52.359432 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:52.359407    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-85b4d8ffd8-jz456"]
Oct 29 22:57:54.164581 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:54.164546    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aff0317-ebc2-46ad-9f61-5e2c550828a0" path="/var/lib/kubelet/pods/6aff0317-ebc2-46ad-9f61-5e2c550828a0/volumes"
Oct 29 22:57:54.165092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:54.165076    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e48dfe-242c-4dc8-8992-398b634ed0d2" path="/var/lib/kubelet/pods/b6e48dfe-242c-4dc8-8992-398b634ed0d2/volumes"
Oct 29 22:57:56.954143 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954108    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"]
Oct 29 22:57:56.954799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954785    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6aff0317-ebc2-46ad-9f61-5e2c550828a0" containerName="payload-processing"
Oct 29 22:57:56.954858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954801    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aff0317-ebc2-46ad-9f61-5e2c550828a0" containerName="payload-processing"
Oct 29 22:57:56.954858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954817    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7af914f4-302d-4a76-9c8c-e68651e7c5d6" containerName="main"
Oct 29 22:57:56.954858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954822    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af914f4-302d-4a76-9c8c-e68651e7c5d6" containerName="main"
Oct 29 22:57:56.954858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954850    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b6e48dfe-242c-4dc8-8992-398b634ed0d2" containerName="payload-pre-processing"
Oct 29 22:57:56.954858 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954856    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e48dfe-242c-4dc8-8992-398b634ed0d2" containerName="payload-pre-processing"
Oct 29 22:57:56.955051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954869    4451 cpu_manager.go:414] "containerMap: removing container" podUID="3494f4d4-87e6-4d73-88b2-2a32d6db7554" containerName="maas-api"
Oct 29 22:57:56.955051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.954873    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3494f4d4-87e6-4d73-88b2-2a32d6db7554" containerName="maas-api"
Oct 29 22:57:56.955051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.955036    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e48dfe-242c-4dc8-8992-398b634ed0d2" containerName="payload-pre-processing"
Oct 29 22:57:56.955051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.955051    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="3494f4d4-87e6-4d73-88b2-2a32d6db7554" containerName="maas-api"
Oct 29 22:57:56.955168 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.955064    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aff0317-ebc2-46ad-9f61-5e2c550828a0" containerName="payload-processing"
Oct 29 22:57:56.955168 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.955072    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af914f4-302d-4a76-9c8c-e68651e7c5d6" containerName="main"
Oct 29 22:57:56.960490 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.960466    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:56.982950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:56.981462    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"]
Oct 29 22:57:57.120744 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.120711    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.120945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.120833    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.120945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.120866    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.120945 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.120891    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.121053 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.120997    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.121091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.121057    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.121181 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.121158    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66s9n\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.121227 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.121213    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.121263 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.121252    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223000 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.222895    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223032    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66s9n\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223105    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223154 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223147    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223239    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223393 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223373    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223426    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223463    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223542 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223498    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.223788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.223759    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.224116 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.224087    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.224217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.224147    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.224217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.224161    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.224217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.224175    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.226272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.226249    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.226351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.226258    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.241211 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.241183    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66s9n\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.241295 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.241276    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.288059 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.288031    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:57.677607 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:57:57.677573    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d30dc2_8472_42f0_ab7c_e4c05bc463f1.slice/crio-ba569fcecdcc4417c675cd1721957af5d750051c50edc190626c17b2d2795900 WatchSource:0}: Error finding container ba569fcecdcc4417c675cd1721957af5d750051c50edc190626c17b2d2795900: Status 404 returned error can't find the container with id ba569fcecdcc4417c675cd1721957af5d750051c50edc190626c17b2d2795900
Oct 29 22:57:57.680691 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.680657    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:57.680797 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.680727    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:57.680797 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.680755    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:57.691071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:57.691051    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"]
Oct 29 22:57:58.343809 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:58.343774    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" event={"ID":"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1","Type":"ContainerStarted","Data":"d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3"}
Oct 29 22:57:58.344310 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:58.343825    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" event={"ID":"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1","Type":"ContainerStarted","Data":"ba569fcecdcc4417c675cd1721957af5d750051c50edc190626c17b2d2795900"}
Oct 29 22:57:58.400004 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:58.399949    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podStartSLOduration=2.399930313 podStartE2EDuration="2.399930313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:58.390122188 +0000 UTC m=+2618.333080886" watchObservedRunningTime="2026-10-29 22:57:58.399930313 +0000 UTC m=+2618.342889026"
Oct 29 22:57:59.288993 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:59.288958    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:57:59.300938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:59.296025    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:57:59.300938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:57:59.296103    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 29 22:58:00.289162 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.289130    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:58:00.289537 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.289190    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 29 22:58:00.841533 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.841497    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"]
Oct 29 22:58:00.846286 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.846264    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.872823 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.872795    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-af260f-openshift-default-dockercfg-glq4f\""
Oct 29 22:58:00.878516 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.878493    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"]
Oct 29 22:58:00.986156 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986123    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x77m4\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986231    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986320 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986306    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986399 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986332    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986485 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986468    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986547 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986536    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986564    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986582    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:00.986671 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:00.986659    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.088895 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.088853    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089101 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.088993    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089101 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089031    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089101 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089093    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089160    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089202    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089259 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089232    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089434 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089335    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089494 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089443    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089507    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089554    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x77m4\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.089668 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.089550    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.117978 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.117345    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.121695 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.121662    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.123268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.122225    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.123268 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.122354    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.125272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.123998    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x77m4\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.138256 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.138228    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.161716 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.161683    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:01.289009 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.288974    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:58:01.289147 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.289034    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 29 22:58:01.527160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.527126    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"]
Oct 29 22:58:01.534271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.534230    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:01.534392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.534323    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:01.534392 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:01.534365    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:02.288856 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:02.288820    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.289077 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:02.288888    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 29 22:58:02.376216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:02.376174    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" event={"ID":"3a29d063-6510-4fe3-8d38-1c24b3a5c70b","Type":"ContainerStarted","Data":"c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07"}
Oct 29 22:58:02.376409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:02.376243    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" event={"ID":"3a29d063-6510-4fe3-8d38-1c24b3a5c70b","Type":"ContainerStarted","Data":"24724aff10d29988c571a3fff4f98ddee9c27a3ed2f9d66eff035553c6efbfd1"}
Oct 29 22:58:02.430018 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:02.428430    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podStartSLOduration=2.428411607 podStartE2EDuration="2.428411607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:02.417073341 +0000 UTC m=+2622.360032074" watchObservedRunningTime="2026-10-29 22:58:02.428411607 +0000 UTC m=+2622.371370313"
Oct 29 22:58:03.162854 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:03.162825    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:03.164715 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:03.164692    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.164818 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:03.164734    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 29 22:58:03.288682 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:03.288646    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.288846 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:03.288708    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 29 22:58:04.164936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:04.163342    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.164936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:04.163405    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 29 22:58:04.289282 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:04.289247    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.289454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:04.289315    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 29 22:58:05.162447 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:05.162406    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:58:05.162630 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:05.162456    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 29 22:58:05.311934 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:05.310730    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:58:05.312444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:05.312105    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:58:05.316206 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:05.316181    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 22:58:06.162111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:06.162078    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:58:06.162278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:06.162134    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 29 22:58:07.161996 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:07.161962    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:58:07.162456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:07.162020    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 29 22:58:08.162134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:08.162100    4451 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:58:08.162612 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:08.162166    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 29 22:58:09.167148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:09.167114    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:09.167559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:09.167534    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:09.168361 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:09.168337    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 22:58:11.044219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.044185    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"]
Oct 29 22:58:11.044676 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.044484    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" podUID="260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" containerName="maas-api" containerID="cri-o://6e85c979e4925c173fca38e2ac383febf353c00fb6feeb18b48fca0b9134a898" gracePeriod=30
Oct 29 22:58:11.521814 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.521776    4451 generic.go:356] "Generic (PLEG): container finished" podID="260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" containerID="6e85c979e4925c173fca38e2ac383febf353c00fb6feeb18b48fca0b9134a898" exitCode=0
Oct 29 22:58:11.522158 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.522116    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" event={"ID":"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43","Type":"ContainerDied","Data":"6e85c979e4925c173fca38e2ac383febf353c00fb6feeb18b48fca0b9134a898"}
Oct 29 22:58:11.522269 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.522199    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m" event={"ID":"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43","Type":"ContainerDied","Data":"e82c3c30628e69b1c4dc059fab0a75148c2f99c97e6ed58ab5d76202c3869cc7"}
Oct 29 22:58:11.522269 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.522225    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e82c3c30628e69b1c4dc059fab0a75148c2f99c97e6ed58ab5d76202c3869cc7"
Oct 29 22:58:11.526662 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.526632    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:58:11.557523 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.557480    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\") pod \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") "
Oct 29 22:58:11.571174 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.564731    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs\") pod \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") "
Oct 29 22:58:11.612454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.612413    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls" pod "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" (UID: "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:11.676948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.650701    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca\") pod \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") "
Oct 29 22:58:11.676948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.650975    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln\" (UniqueName: \"kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln\") pod \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\" (UID: \"260a2e7f-c3a3-4735-9bdb-b5b79fa0de43\") "
Oct 29 22:58:11.676948 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.651841    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.719939 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.697033    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca" pod "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" (UID: "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:11.755012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.750664    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln" pod "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" (UID: "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43"). InnerVolumeSpecName "kube-api-access-xdjln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:11.755012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.754643    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjln\" (UniqueName: \"kubernetes.io/projected/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-kube-api-access-xdjln\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.755012 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.754673    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.757986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.756392    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs" pod "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" (UID: "260a2e7f-c3a3-4735-9bdb-b5b79fa0de43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:11.856307 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:11.856257    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.528510 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:12.528427    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"
Oct 29 22:58:12.573522 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:12.573488    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"]
Oct 29 22:58:12.583332 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:12.583307    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a962d7-6b5d9bf4d5-dwg6m"]
Oct 29 22:58:14.164953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:14.164897    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" path="/var/lib/kubelet/pods/260a2e7f-c3a3-4735-9bdb-b5b79fa0de43/volumes"
Oct 29 22:58:18.746171 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:58:18.746136    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:58:27.609645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:27.609608    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"]
Oct 29 22:58:42.634598 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:42.634483    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw" podUID="8e971f33-1553-4029-962f-4db5627aad5b" containerName="main" containerID="cri-o://bb8247a2141f7fc8d2136bcf5f7e5670de599d52cbb40ee176f6c64d80a98584" gracePeriod=45
Oct 29 22:58:42.816461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:42.816423    4451 generic.go:356] "Generic (PLEG): container finished" podID="8e971f33-1553-4029-962f-4db5627aad5b" containerID="bb8247a2141f7fc8d2136bcf5f7e5670de599d52cbb40ee176f6c64d80a98584" exitCode=0
Oct 29 22:58:42.824073 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:42.824031    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw" event={"ID":"8e971f33-1553-4029-962f-4db5627aad5b","Type":"ContainerDied","Data":"bb8247a2141f7fc8d2136bcf5f7e5670de599d52cbb40ee176f6c64d80a98584"}
Oct 29 22:58:43.168031 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.166570    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:58:43.394950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.364448    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g\" (UniqueName: \"kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g\") pod \"8e971f33-1553-4029-962f-4db5627aad5b\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") "
Oct 29 22:58:43.394950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.364531    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home\") pod \"8e971f33-1553-4029-962f-4db5627aad5b\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") "
Oct 29 22:58:43.394950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.364659    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs\") pod \"8e971f33-1553-4029-962f-4db5627aad5b\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") "
Oct 29 22:58:43.394950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.364772    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir\") pod \"8e971f33-1553-4029-962f-4db5627aad5b\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") "
Oct 29 22:58:43.394950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.364883    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache\") pod \"8e971f33-1553-4029-962f-4db5627aad5b\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") "
Oct 29 22:58:43.394950 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.365029    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm\") pod \"8e971f33-1553-4029-962f-4db5627aad5b\" (UID: \"8e971f33-1553-4029-962f-4db5627aad5b\") "
Oct 29 22:58:43.415999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.396176    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home" pod "8e971f33-1553-4029-962f-4db5627aad5b" (UID: "8e971f33-1553-4029-962f-4db5627aad5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:43.415999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.407386    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache" pod "8e971f33-1553-4029-962f-4db5627aad5b" (UID: "8e971f33-1553-4029-962f-4db5627aad5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:43.415999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.407474    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm" pod "8e971f33-1553-4029-962f-4db5627aad5b" (UID: "8e971f33-1553-4029-962f-4db5627aad5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:43.415999 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.409102    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g" pod "8e971f33-1553-4029-962f-4db5627aad5b" (UID: "8e971f33-1553-4029-962f-4db5627aad5b"). InnerVolumeSpecName "kube-api-access-wtg5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:43.462937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.453031    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir" pod "8e971f33-1553-4029-962f-4db5627aad5b" (UID: "8e971f33-1553-4029-962f-4db5627aad5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:43.478412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.478363    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:43.478412 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.478409    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:43.478674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.478424    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:43.478674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.478437    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtg5g\" (UniqueName: \"kubernetes.io/projected/8e971f33-1553-4029-962f-4db5627aad5b-kube-api-access-wtg5g\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:43.478674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.478449    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e971f33-1553-4029-962f-4db5627aad5b-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:43.586960 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.586328    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs" pod "8e971f33-1553-4029-962f-4db5627aad5b" (UID: "8e971f33-1553-4029-962f-4db5627aad5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:43.687371 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.687317    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e971f33-1553-4029-962f-4db5627aad5b-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:58:43.835220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.834952    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"
Oct 29 22:58:43.835220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.834976    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw" event={"ID":"8e971f33-1553-4029-962f-4db5627aad5b","Type":"ContainerDied","Data":"10a382f1e336a38101c0f6d6a7503020f24aff2ad1925332f3bf4d6918676947"}
Oct 29 22:58:43.835220 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.835035    4451 scope.go:122] "RemoveContainer" containerID="bb8247a2141f7fc8d2136bcf5f7e5670de599d52cbb40ee176f6c64d80a98584"
Oct 29 22:58:43.886216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.886189    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"]
Oct 29 22:58:43.894713 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:43.894691    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-af260f/sub-test-model-af260f-kserve-648f545ccf-4svlw"]
Oct 29 22:58:44.164839 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:44.164800    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e971f33-1553-4029-962f-4db5627aad5b" path="/var/lib/kubelet/pods/8e971f33-1553-4029-962f-4db5627aad5b/volumes"
Oct 29 22:58:46.639791 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:46.639754    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"]
Oct 29 22:58:48.748714 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:58:48.748665    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:58:55.507002 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:58:55.499330    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"]
Oct 29 22:59:01.668335 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:01.668265    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" podUID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" containerName="main" containerID="cri-o://65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4" gracePeriod=45
Oct 29 22:59:01.739686 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:01.739653    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" podUID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": dial tcp 10.133.0.139:8000: connect: connection refused"
Oct 29 22:59:01.950862 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:01.950832    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:59:02.074546 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.074513    4451 generic.go:356] "Generic (PLEG): container finished" podID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" containerID="65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4" exitCode=0
Oct 29 22:59:02.074729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.074566    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"
Oct 29 22:59:02.074729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.074603    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" event={"ID":"ae3d453c-44b8-49df-ac0a-3d8c539b20c3","Type":"ContainerDied","Data":"65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4"}
Oct 29 22:59:02.074729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.074650    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp" event={"ID":"ae3d453c-44b8-49df-ac0a-3d8c539b20c3","Type":"ContainerDied","Data":"0ec77347b89766e49a4f0a4eaf3ad303cbee640bd680707500dfdea4f2df3654"}
Oct 29 22:59:02.074729 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.074668    4451 scope.go:122] "RemoveContainer" containerID="65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4"
Oct 29 22:59:02.084819 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.084803    4451 scope.go:122] "RemoveContainer" containerID="65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4"
Oct 29 22:59:02.085122 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:02.085089    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4\": container with ID starting with 65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4 not found: ID does not exist" containerID="65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4"
Oct 29 22:59:02.085219 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.085125    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4"} err="failed to get container status \"65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4\": rpc error: code = NotFound desc = could not find container \"65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4\": container with ID starting with 65289d16dac384d20a2075a1afcef87de824ba83d6052bb31ccff6a7e99967f4 not found: ID does not exist"
Oct 29 22:59:02.107179 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107153    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir\") pod \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") "
Oct 29 22:59:02.107272 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107214    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs\") pod \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") "
Oct 29 22:59:02.107314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107281    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache\") pod \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") "
Oct 29 22:59:02.107358 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107324    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home\") pod \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") "
Oct 29 22:59:02.107420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107408    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm\") pod \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") "
Oct 29 22:59:02.107470 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107445    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir" pod "ae3d453c-44b8-49df-ac0a-3d8c539b20c3" (UID: "ae3d453c-44b8-49df-ac0a-3d8c539b20c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:02.107545 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107531    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j\" (UniqueName: \"kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j\") pod \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\" (UID: \"ae3d453c-44b8-49df-ac0a-3d8c539b20c3\") "
Oct 29 22:59:02.107615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107550    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache" pod "ae3d453c-44b8-49df-ac0a-3d8c539b20c3" (UID: "ae3d453c-44b8-49df-ac0a-3d8c539b20c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:02.107615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.107594    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home" pod "ae3d453c-44b8-49df-ac0a-3d8c539b20c3" (UID: "ae3d453c-44b8-49df-ac0a-3d8c539b20c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:02.108200 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.108179    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:02.108314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.108216    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:02.108314 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.108229    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:02.109296 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.109275    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs" pod "ae3d453c-44b8-49df-ac0a-3d8c539b20c3" (UID: "ae3d453c-44b8-49df-ac0a-3d8c539b20c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:02.109543 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.109524    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm" pod "ae3d453c-44b8-49df-ac0a-3d8c539b20c3" (UID: "ae3d453c-44b8-49df-ac0a-3d8c539b20c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:02.109667 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.109652    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j" pod "ae3d453c-44b8-49df-ac0a-3d8c539b20c3" (UID: "ae3d453c-44b8-49df-ac0a-3d8c539b20c3"). InnerVolumeSpecName "kube-api-access-ndd7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:02.210091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.210024    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndd7j\" (UniqueName: \"kubernetes.io/projected/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-kube-api-access-ndd7j\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:02.210091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.210051    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:02.210091 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.210063    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3d453c-44b8-49df-ac0a-3d8c539b20c3-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:02.411458 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.411430    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"]
Oct 29 22:59:02.415105 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:02.415079    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d68c2c/sub-test-model-d68c2c-kserve-6fdd6d6ffc-5jxcp"]
Oct 29 22:59:03.033937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.011582    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"]
Oct 29 22:59:03.048943 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.034737    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" podUID="b62cafb1-4b88-4125-91ae-72b574f847f3" containerName="maas-api" containerID="cri-o://358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7" gracePeriod=30
Oct 29 22:59:03.748558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.748528    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:59:03.870585 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.870543    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca\") pod \"b62cafb1-4b88-4125-91ae-72b574f847f3\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") "
Oct 29 22:59:03.870788 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.870617    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm\" (UniqueName: \"kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm\") pod \"b62cafb1-4b88-4125-91ae-72b574f847f3\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") "
Oct 29 22:59:03.871035 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.871006    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs\") pod \"b62cafb1-4b88-4125-91ae-72b574f847f3\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") "
Oct 29 22:59:03.871151 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.871062    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\") pod \"b62cafb1-4b88-4125-91ae-72b574f847f3\" (UID: \"b62cafb1-4b88-4125-91ae-72b574f847f3\") "
Oct 29 22:59:03.872795 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.872763    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca" pod "b62cafb1-4b88-4125-91ae-72b574f847f3" (UID: "b62cafb1-4b88-4125-91ae-72b574f847f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:03.877069 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.876991    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm" pod "b62cafb1-4b88-4125-91ae-72b574f847f3" (UID: "b62cafb1-4b88-4125-91ae-72b574f847f3"). InnerVolumeSpecName "kube-api-access-gvrvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:03.878407 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.878248    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs" pod "b62cafb1-4b88-4125-91ae-72b574f847f3" (UID: "b62cafb1-4b88-4125-91ae-72b574f847f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:03.880967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.880934    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls" pod "b62cafb1-4b88-4125-91ae-72b574f847f3" (UID: "b62cafb1-4b88-4125-91ae-72b574f847f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:03.973936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.973834    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:03.973936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.973866    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62cafb1-4b88-4125-91ae-72b574f847f3-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:03.973936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.973879    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62cafb1-4b88-4125-91ae-72b574f847f3-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:03.973936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:03.973892    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvrvm\" (UniqueName: \"kubernetes.io/projected/b62cafb1-4b88-4125-91ae-72b574f847f3-kube-api-access-gvrvm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.165674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.165568    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" path="/var/lib/kubelet/pods/ae3d453c-44b8-49df-ac0a-3d8c539b20c3/volumes"
Oct 29 22:59:04.204867 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.204826    4451 generic.go:356] "Generic (PLEG): container finished" podID="b62cafb1-4b88-4125-91ae-72b574f847f3" containerID="358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7" exitCode=0
Oct 29 22:59:04.205072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.204877    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"
Oct 29 22:59:04.205072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.204878    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" event={"ID":"b62cafb1-4b88-4125-91ae-72b574f847f3","Type":"ContainerDied","Data":"358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7"}
Oct 29 22:59:04.205072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.204981    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz" event={"ID":"b62cafb1-4b88-4125-91ae-72b574f847f3","Type":"ContainerDied","Data":"0d7533f520062462b25d759a023d91b3e9827317117d6e1d60eac5362c9318f9"}
Oct 29 22:59:04.205072 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.205015    4451 scope.go:122] "RemoveContainer" containerID="358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7"
Oct 29 22:59:04.216064 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.216046    4451 scope.go:122] "RemoveContainer" containerID="358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7"
Oct 29 22:59:04.216325 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:04.216307    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7\": container with ID starting with 358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7 not found: ID does not exist" containerID="358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7"
Oct 29 22:59:04.216386 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.216335    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7"} err="failed to get container status \"358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7\": rpc error: code = NotFound desc = could not find container \"358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7\": container with ID starting with 358c31725543a7ba5475021c3f0bc3ba553398a5958d8509eaa91bb5261377f7 not found: ID does not exist"
Oct 29 22:59:04.254475 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.254405    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"]
Oct 29 22:59:04.264026 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:04.263994    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4796a9-5fbc8847bb-jfzdz"]
Oct 29 22:59:06.166029 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:06.165998    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62cafb1-4b88-4125-91ae-72b574f847f3" path="/var/lib/kubelet/pods/b62cafb1-4b88-4125-91ae-72b574f847f3/volumes"
Oct 29 22:59:10.744955 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:10.744882    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg" podUID="8be302e7-6277-4830-99a2-654c81400255" containerName="main" containerID="cri-o://a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9" gracePeriod=45
Oct 29 22:59:11.074134 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.074097    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:59:11.124346 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124310    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk\" (UniqueName: \"kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk\") pod \"8be302e7-6277-4830-99a2-654c81400255\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") "
Oct 29 22:59:11.124509 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124428    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir\") pod \"8be302e7-6277-4830-99a2-654c81400255\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") "
Oct 29 22:59:11.124560 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124511    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs\" (UniqueName: \"kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs\") pod \"8be302e7-6277-4830-99a2-654c81400255\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") "
Oct 29 22:59:11.124560 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124551    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home\") pod \"8be302e7-6277-4830-99a2-654c81400255\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") "
Oct 29 22:59:11.124752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124735    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm\") pod \"8be302e7-6277-4830-99a2-654c81400255\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") "
Oct 29 22:59:11.124852 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124766    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache\") pod \"8be302e7-6277-4830-99a2-654c81400255\" (UID: \"8be302e7-6277-4830-99a2-654c81400255\") "
Oct 29 22:59:11.124952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124734    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir" pod "8be302e7-6277-4830-99a2-654c81400255" (UID: "8be302e7-6277-4830-99a2-654c81400255"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:11.124952 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.124866    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home" pod "8be302e7-6277-4830-99a2-654c81400255" (UID: "8be302e7-6277-4830-99a2-654c81400255"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:11.125280 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.125252    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache" pod "8be302e7-6277-4830-99a2-654c81400255" (UID: "8be302e7-6277-4830-99a2-654c81400255"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:11.125454 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.125428    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:11.125527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.125461    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:11.125527 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.125473    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:11.127148 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.127119    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk" pod "8be302e7-6277-4830-99a2-654c81400255" (UID: "8be302e7-6277-4830-99a2-654c81400255"). InnerVolumeSpecName "kube-api-access-hlhmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:11.127309 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.127287    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm" pod "8be302e7-6277-4830-99a2-654c81400255" (UID: "8be302e7-6277-4830-99a2-654c81400255"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:11.127469 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.127453    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs" pod "8be302e7-6277-4830-99a2-654c81400255" (UID: "8be302e7-6277-4830-99a2-654c81400255"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:11.227986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.227950    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be302e7-6277-4830-99a2-654c81400255-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:11.227986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.227981    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be302e7-6277-4830-99a2-654c81400255-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:11.227986 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.227991    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhmk\" (UniqueName: \"kubernetes.io/projected/8be302e7-6277-4830-99a2-654c81400255-kube-api-access-hlhmk\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:11.370864 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.370825    4451 generic.go:356] "Generic (PLEG): container finished" podID="8be302e7-6277-4830-99a2-654c81400255" containerID="a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9" exitCode=0
Oct 29 22:59:11.371127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.370970    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg" event={"ID":"8be302e7-6277-4830-99a2-654c81400255","Type":"ContainerDied","Data":"a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9"}
Oct 29 22:59:11.371127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.371005    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg" event={"ID":"8be302e7-6277-4830-99a2-654c81400255","Type":"ContainerDied","Data":"7cfe8e6b65c88818875611ff804e2964147dccb865aff6e7e7aeeee0ee38948a"}
Oct 29 22:59:11.371127 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.371024    4451 scope.go:122] "RemoveContainer" containerID="a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9"
Oct 29 22:59:11.371333 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.371164    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"
Oct 29 22:59:11.409092 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.409061    4451 scope.go:122] "RemoveContainer" containerID="a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9"
Oct 29 22:59:11.409564 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:11.409531    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9\": container with ID starting with a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9 not found: ID does not exist" containerID="a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9"
Oct 29 22:59:11.409681 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.409578    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9"} err="failed to get container status \"a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9\": rpc error: code = NotFound desc = could not find container \"a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9\": container with ID starting with a9e29a06a69fe8a43cae414260c55fd1c9b0b63bd8368dd937a6620afea504c9 not found: ID does not exist"
Oct 29 22:59:11.446185 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.446007    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"]
Oct 29 22:59:11.454967 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:11.451800    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f8ca96/rate-test-model-f8ca96-kserve-586cd65bd5-vctzg"]
Oct 29 22:59:12.165887 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:12.165854    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be302e7-6277-4830-99a2-654c81400255" path="/var/lib/kubelet/pods/8be302e7-6277-4830-99a2-654c81400255/volumes"
Oct 29 22:59:18.745439 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:18.745403    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:59:21.074582 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.074523    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"]
Oct 29 22:59:21.621684 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.621634    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"]
Oct 29 22:59:21.622693 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622661    4451 cpu_manager.go:414] "containerMap: removing container" podUID="260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" containerName="maas-api"
Oct 29 22:59:21.622693 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622687    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" containerName="maas-api"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622709    4451 cpu_manager.go:414] "containerMap: removing container" podUID="8be302e7-6277-4830-99a2-654c81400255" containerName="main"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622715    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be302e7-6277-4830-99a2-654c81400255" containerName="main"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622747    4451 cpu_manager.go:414] "containerMap: removing container" podUID="8e971f33-1553-4029-962f-4db5627aad5b" containerName="main"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622752    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e971f33-1553-4029-962f-4db5627aad5b" containerName="main"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622760    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" containerName="main"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622765    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" containerName="main"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622772    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b62cafb1-4b88-4125-91ae-72b574f847f3" containerName="maas-api"
Oct 29 22:59:21.622872 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622776    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62cafb1-4b88-4125-91ae-72b574f847f3" containerName="maas-api"
Oct 29 22:59:21.623303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622878    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e971f33-1553-4029-962f-4db5627aad5b" containerName="main"
Oct 29 22:59:21.623303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622888    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3d453c-44b8-49df-ac0a-3d8c539b20c3" containerName="main"
Oct 29 22:59:21.623303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622893    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62cafb1-4b88-4125-91ae-72b574f847f3" containerName="maas-api"
Oct 29 22:59:21.623303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622902    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="260a2e7f-c3a3-4735-9bdb-b5b79fa0de43" containerName="maas-api"
Oct 29 22:59:21.623303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.622925    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be302e7-6277-4830-99a2-654c81400255" containerName="main"
Oct 29 22:59:21.627786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.627763    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.649410 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.649381    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7be71d\"/\"openshift-service-ca.crt\""
Oct 29 22:59:21.649618 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.649604    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7be71d\"/\"default-dockercfg-56h99\""
Oct 29 22:59:21.649683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.649620    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7be71d\"/\"kube-root-ca.crt\""
Oct 29 22:59:21.649683 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.649675    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7be71d\"/\"ipp-route-7be71d-kserve-self-signed-certs\""
Oct 29 22:59:21.667706 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.667674    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"]
Oct 29 22:59:21.705182 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.705132    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.705377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.705195    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.705377 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.705297    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.705477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.705388    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.705477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.705441    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmwb\" (UniqueName: \"kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.705563 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.705545    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807347 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807306    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807374    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmwb\" (UniqueName: \"kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807559 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807476    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807708 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807687    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807697    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807771 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807744    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807891 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807785    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.807973 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.807938    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.808141 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.808121    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.810132 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.810101    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.810417 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.810400    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.847629 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.847596    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmwb\" (UniqueName: \"kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb\") pod \"ipp-route-7be71d-kserve-bfd576cf7-n8tl5\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") " pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:21.946987 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:21.946055    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:22.387521 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:22.387489    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"]
Oct 29 22:59:22.399171 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:59:22.398064    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbf17621_84f3_4c2c_a7a6_b41679a95a08.slice/crio-699efcf615aadb5881452b83216c2930149e0bd0b3161796c6b3635ed50eb138 WatchSource:0}: Error finding container 699efcf615aadb5881452b83216c2930149e0bd0b3161796c6b3635ed50eb138: Status 404 returned error can't find the container with id 699efcf615aadb5881452b83216c2930149e0bd0b3161796c6b3635ed50eb138
Oct 29 22:59:22.544586 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:22.544549    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5" event={"ID":"fbf17621-84f3-4c2c-a7a6-b41679a95a08","Type":"ContainerStarted","Data":"8295eb49b03819d39f187384e434fff3ac33042e25de2c0ab501a767885a476e"}
Oct 29 22:59:22.544754 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:22.544597    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5" event={"ID":"fbf17621-84f3-4c2c-a7a6-b41679a95a08","Type":"ContainerStarted","Data":"699efcf615aadb5881452b83216c2930149e0bd0b3161796c6b3635ed50eb138"}
Oct 29 22:59:22.592574 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:22.592499    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5" podStartSLOduration=1.5924789050000001 podStartE2EDuration="1.592478905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:22.588725811 +0000 UTC m=+2702.531684517" watchObservedRunningTime="2026-10-29 22:59:22.592478905 +0000 UTC m=+2702.535437702"
Oct 29 22:59:22.946798 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:22.946764    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:31.764877 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:31.764837    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"]
Oct 29 22:59:31.765558 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:31.765080    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy" containerID="cri-o://9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21" gracePeriod=30
Oct 29 22:59:31.947486 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:31.947428    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:31.960456 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:31.960420    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:32.635501 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:32.635470    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 22:59:36.171051 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.171006    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv" podUID="6639a77b-8158-4f16-bb65-fc0aa80b259f" containerName="main" containerID="cri-o://630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750" gracePeriod=45
Oct 29 22:59:36.444294 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.444226    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:59:36.572937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.564598    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home\") pod \"6639a77b-8158-4f16-bb65-fc0aa80b259f\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") "
Oct 29 22:59:36.572937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.564669    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir\") pod \"6639a77b-8158-4f16-bb65-fc0aa80b259f\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") "
Oct 29 22:59:36.572937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.564814    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm\") pod \"6639a77b-8158-4f16-bb65-fc0aa80b259f\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") "
Oct 29 22:59:36.572937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.564881    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl\" (UniqueName: \"kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl\") pod \"6639a77b-8158-4f16-bb65-fc0aa80b259f\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") "
Oct 29 22:59:36.572937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.565002    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs\") pod \"6639a77b-8158-4f16-bb65-fc0aa80b259f\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") "
Oct 29 22:59:36.572937 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.565057    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache\") pod \"6639a77b-8158-4f16-bb65-fc0aa80b259f\" (UID: \"6639a77b-8158-4f16-bb65-fc0aa80b259f\") "
Oct 29 22:59:36.581933 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.575814    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm" pod "6639a77b-8158-4f16-bb65-fc0aa80b259f" (UID: "6639a77b-8158-4f16-bb65-fc0aa80b259f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:36.586621 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.586587    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home" pod "6639a77b-8158-4f16-bb65-fc0aa80b259f" (UID: "6639a77b-8158-4f16-bb65-fc0aa80b259f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:36.604126 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.604090    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir" pod "6639a77b-8158-4f16-bb65-fc0aa80b259f" (UID: "6639a77b-8158-4f16-bb65-fc0aa80b259f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:36.638752 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.638715    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl" pod "6639a77b-8158-4f16-bb65-fc0aa80b259f" (UID: "6639a77b-8158-4f16-bb65-fc0aa80b259f"). InnerVolumeSpecName "kube-api-access-nhbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:36.647555 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.647520    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache" pod "6639a77b-8158-4f16-bb65-fc0aa80b259f" (UID: "6639a77b-8158-4f16-bb65-fc0aa80b259f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:36.667390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.667356    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs" pod "6639a77b-8158-4f16-bb65-fc0aa80b259f" (UID: "6639a77b-8158-4f16-bb65-fc0aa80b259f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:36.667390 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.667370    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:36.667734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.667416    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:36.667734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.667426    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:36.667734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.667436    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhbbl\" (UniqueName: \"kubernetes.io/projected/6639a77b-8158-4f16-bb65-fc0aa80b259f-kube-api-access-nhbbl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:36.667734 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.667451    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6639a77b-8158-4f16-bb65-fc0aa80b259f-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:36.673968 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.673933    4451 generic.go:356] "Generic (PLEG): container finished" podID="6639a77b-8158-4f16-bb65-fc0aa80b259f" containerID="630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750" exitCode=0
Oct 29 22:59:36.674167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.674018    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv" event={"ID":"6639a77b-8158-4f16-bb65-fc0aa80b259f","Type":"ContainerDied","Data":"630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750"}
Oct 29 22:59:36.674167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.674058    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv" event={"ID":"6639a77b-8158-4f16-bb65-fc0aa80b259f","Type":"ContainerDied","Data":"7e94de05ce4d92edb28778670b24ad8e96b5df1b344a6cb3321e9ff355ed8674"}
Oct 29 22:59:36.674167 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.674082    4451 scope.go:122] "RemoveContainer" containerID="630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750"
Oct 29 22:59:36.674461 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.674257    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"
Oct 29 22:59:36.699385 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.699359    4451 scope.go:122] "RemoveContainer" containerID="630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750"
Oct 29 22:59:36.699964 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:36.699679    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750\": container with ID starting with 630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750 not found: ID does not exist" containerID="630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750"
Oct 29 22:59:36.699964 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.699732    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750"} err="failed to get container status \"630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750\": rpc error: code = NotFound desc = could not find container \"630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750\": container with ID starting with 630c772b7ee572bd5f8b4252a52f9e1bfd0e91aa4a530f53f5f3c270eb121750 not found: ID does not exist"
Oct 29 22:59:36.730786 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.730760    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"]
Oct 29 22:59:36.738469 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.738306    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c933cb/rate-test-model-c933cb-kserve-54bfb69d9c-7rqmv"]
Oct 29 22:59:36.769420 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:36.769395    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6639a77b-8158-4f16-bb65-fc0aa80b259f-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.040444 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.040415    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:59:37.074037 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074011    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074051    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074079    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074171    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074217 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074210    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074287    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074348    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074450 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074327    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:37.074651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074446    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:37.074651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074480    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:37.074651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074564    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074651 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074640    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs\") pod \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\" (UID: \"00e166b0-1cc0-44e8-821a-c1fddbf98fc6\") "
Oct 29 22:59:37.074953 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.074905    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:37.075151 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.075118    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:37.075424 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.075404    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.075506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.075430    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.075506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.075445    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.075506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.075457    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.075506 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.075469    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.076529 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.076501    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "kube-api-access-6z2wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:37.076645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.076629    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:37.076972 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.076949    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:37.077045 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.076985    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy" pod "00e166b0-1cc0-44e8-821a-c1fddbf98fc6" (UID: "00e166b0-1cc0-44e8-821a-c1fddbf98fc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:37.177402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.177369    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.177402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.177396    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.177402 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.177405    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.177842 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.177417    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z2wr\" (UniqueName: \"kubernetes.io/projected/00e166b0-1cc0-44e8-821a-c1fddbf98fc6-kube-api-access-6z2wr\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:37.693586 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.693550    4451 generic.go:356] "Generic (PLEG): container finished" podID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerID="9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21" exitCode=0
Oct 29 22:59:37.693761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.693601    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" event={"ID":"00e166b0-1cc0-44e8-821a-c1fddbf98fc6","Type":"ContainerDied","Data":"9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21"}
Oct 29 22:59:37.693761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.693628    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"
Oct 29 22:59:37.693761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.693672    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf" event={"ID":"00e166b0-1cc0-44e8-821a-c1fddbf98fc6","Type":"ContainerDied","Data":"a8deefd43d5dc5b7f751193987c31d862933ed866784cad92911f4b468130a13"}
Oct 29 22:59:37.693761 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.693696    4451 scope.go:122] "RemoveContainer" containerID="9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21"
Oct 29 22:59:37.704758 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.704740    4451 scope.go:122] "RemoveContainer" containerID="9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21"
Oct 29 22:59:37.705047 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:37.705026    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21\": container with ID starting with 9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21 not found: ID does not exist" containerID="9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21"
Oct 29 22:59:37.705111 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.705059    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21"} err="failed to get container status \"9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21\": rpc error: code = NotFound desc = could not find container \"9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21\": container with ID starting with 9fd64125288a4af3daa67c613b8f4099a164f1cf47dbaf798182a76afcc46d21 not found: ID does not exist"
Oct 29 22:59:37.749324 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.749301    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"]
Oct 29 22:59:37.759306 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:37.759276    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-2t2kf"]
Oct 29 22:59:38.306936 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:38.306417    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" path="/var/lib/kubelet/pods/00e166b0-1cc0-44e8-821a-c1fddbf98fc6/volumes"
Oct 29 22:59:38.309994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:38.309052    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6639a77b-8158-4f16-bb65-fc0aa80b259f" path="/var/lib/kubelet/pods/6639a77b-8158-4f16-bb65-fc0aa80b259f/volumes"
Oct 29 22:59:40.201043 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:40.192660    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"]
Oct 29 22:59:40.379977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:40.372574    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"]
Oct 29 22:59:42.091186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.091149    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"]
Oct 29 22:59:42.092015 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.091995    4451 cpu_manager.go:414] "containerMap: removing container" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy"
Oct 29 22:59:42.092071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.092016    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy"
Oct 29 22:59:42.092071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.092031    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6639a77b-8158-4f16-bb65-fc0aa80b259f" containerName="main"
Oct 29 22:59:42.092071 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.092037    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6639a77b-8158-4f16-bb65-fc0aa80b259f" containerName="main"
Oct 29 22:59:42.092221 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.092203    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e166b0-1cc0-44e8-821a-c1fddbf98fc6" containerName="istio-proxy"
Oct 29 22:59:42.092298 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.092224    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6639a77b-8158-4f16-bb65-fc0aa80b259f" containerName="main"
Oct 29 22:59:42.097504 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.097481    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.109304 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.109273    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.109488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.109373    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tb9n\" (UniqueName: \"kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.109488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.109432    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.109488 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.109457    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.117463 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.117436    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-80248b-serving-cert\""
Oct 29 22:59:42.122603 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.122578    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"]
Oct 29 22:59:42.177057 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.177025    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"]
Oct 29 22:59:42.182966 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.182892    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.201357 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.201322    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"]
Oct 29 22:59:42.210832 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.210804    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.210994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.210867    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.210994 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.210906    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.211117 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211089    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.211368 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211341    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.211492 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211390    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.211492 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211445    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.211613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211516    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tb9n\" (UniqueName: \"kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.211613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211543    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.211613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211570    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.211613 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.211589    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggmn\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.211873 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:42.211855    4451 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-serving-cert: secret "maas-api-e2e-ipp-a-80248b-serving-cert" not found
Oct 29 22:59:42.211987 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:42.211973    4451 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls podName:7ef68324-fbef-4b8a-896e-a5fd11541e4f nodeName:}" failed. No retries permitted until 2026-10-29 22:59:42.711905181 +0000 UTC m=+2722.654863879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls") pod "maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" (UID: "7ef68324-fbef-4b8a-896e-a5fd11541e4f") : secret "maas-api-e2e-ipp-a-80248b-serving-cert" not found
Oct 29 22:59:42.212977 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.212415    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.215278 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.215251    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.217303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.217282    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:59:42.217303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.217292    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:59:42.217583 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.217562    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:59:42.217658 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.217630    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rgwvk\""
Oct 29 22:59:42.243391 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.243105    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tb9n\" (UniqueName: \"kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.314216 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.313662    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.314507 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.314479    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.316186 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.314820    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.316445 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.316422    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggmn\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.316615 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.316596    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.316756 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.316739    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.316958 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.316939    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.319326 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.319301    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.321676 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.321647    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.323433 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.323385    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.324471 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.324449    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.324674 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.324657    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.342285 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.342215    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggmn\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.343351 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.343329    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9jn89\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.500645 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.500606    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:42.723481 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.723395    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.725969 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.725899    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:42.833173 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:59:42.833139    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f36b40c_e687_4e2f_a34b_1d483b80ead5.slice/crio-14e74e6728a3aa165a454496e0667542f617de16298e02c55d5933bdd2c18da6 WatchSource:0}: Error finding container 14e74e6728a3aa165a454496e0667542f617de16298e02c55d5933bdd2c18da6: Status 404 returned error can't find the container with id 14e74e6728a3aa165a454496e0667542f617de16298e02c55d5933bdd2c18da6
Oct 29 22:59:42.833349 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.833327    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"]
Oct 29 22:59:42.836160 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.836128    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:42.836271 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:42.836204    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:43.014150 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.014057    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:43.341543 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.341514    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"]
Oct 29 22:59:43.346925 ip-10-0-143-15 kubenswrapper[4451]: W1029 22:59:43.346880    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef68324_fbef_4b8a_896e_a5fd11541e4f.slice/crio-1af7f558515dec48d97198a9dbfb4af951c1c31f112f95ac67d1dcd8b1a22c20 WatchSource:0}: Error finding container 1af7f558515dec48d97198a9dbfb4af951c1c31f112f95ac67d1dcd8b1a22c20: Status 404 returned error can't find the container with id 1af7f558515dec48d97198a9dbfb4af951c1c31f112f95ac67d1dcd8b1a22c20
Oct 29 22:59:43.348409 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.348385    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:59:43.789326 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.789293    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" event={"ID":"7ef68324-fbef-4b8a-896e-a5fd11541e4f","Type":"ContainerStarted","Data":"1af7f558515dec48d97198a9dbfb4af951c1c31f112f95ac67d1dcd8b1a22c20"}
Oct 29 22:59:43.794931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.792264    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" event={"ID":"3f36b40c-e687-4e2f-a34b-1d483b80ead5","Type":"ContainerStarted","Data":"52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a"}
Oct 29 22:59:43.794931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.792315    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" event={"ID":"3f36b40c-e687-4e2f-a34b-1d483b80ead5","Type":"ContainerStarted","Data":"14e74e6728a3aa165a454496e0667542f617de16298e02c55d5933bdd2c18da6"}
Oct 29 22:59:43.794931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.792771    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:43.794931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.794747    4451 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9jn89 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:59:43.794931 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.794792    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" podUID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:59:43.851030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:43.850958    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" podStartSLOduration=1.8509412969999999 podStartE2EDuration="1.850941297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:43.840598632 +0000 UTC m=+2723.783557339" watchObservedRunningTime="2026-10-29 22:59:43.850941297 +0000 UTC m=+2723.793900003"
Oct 29 22:59:44.801981 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:44.801895    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" event={"ID":"7ef68324-fbef-4b8a-896e-a5fd11541e4f","Type":"ContainerStarted","Data":"2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750"}
Oct 29 22:59:44.802386 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:44.802227    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:44.802838 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:44.802814    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 22:59:44.843874 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:44.843824    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" podStartSLOduration=2.503305235 podStartE2EDuration="2.843812088s" totalImagesPullingTime="340.506853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:44.843585842 +0000 UTC m=+2724.786544546" watchObservedRunningTime="2026-10-29 22:59:44.843812088 +0000 UTC m=+2724.786770792"
Oct 29 22:59:48.742515 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:48.742474    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 22:59:50.815095 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:50.815063    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 22:59:55.445503 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.445459    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main" containerID="cri-o://db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1" gracePeriod=45
Oct 29 22:59:55.531030 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.530975    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd" podUID="e0dd0998-0241-4637-b452-e3146fd29b32" containerName="main" containerID="cri-o://f7537e08e4c0a7d1d4c31b5c507ef7c76bd4dd606091ecd58e043a5e5e1a55c0" gracePeriod=45
Oct 29 22:59:55.848971 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.848925    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:59:55.884884 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.884857    4451 generic.go:356] "Generic (PLEG): container finished" podID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerID="db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1" exitCode=0
Oct 29 22:59:55.885028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.884897    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"
Oct 29 22:59:55.885028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.884973    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" event={"ID":"bb31020f-0dc3-40f9-9f52-72462ba65bdc","Type":"ContainerDied","Data":"db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1"}
Oct 29 22:59:55.885028 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.885024    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" event={"ID":"bb31020f-0dc3-40f9-9f52-72462ba65bdc","Type":"ContainerDied","Data":"860409a1a34a093222140a90c1f811930c67607708979a6050c93f4d66b44f00"}
Oct 29 22:59:55.885202 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.885046    4451 scope.go:122] "RemoveContainer" containerID="db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1"
Oct 29 22:59:55.886405 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.886384    4451 generic.go:356] "Generic (PLEG): container finished" podID="e0dd0998-0241-4637-b452-e3146fd29b32" containerID="f7537e08e4c0a7d1d4c31b5c507ef7c76bd4dd606091ecd58e043a5e5e1a55c0" exitCode=0
Oct 29 22:59:55.886525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.886412    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd" event={"ID":"e0dd0998-0241-4637-b452-e3146fd29b32","Type":"ContainerDied","Data":"f7537e08e4c0a7d1d4c31b5c507ef7c76bd4dd606091ecd58e043a5e5e1a55c0"}
Oct 29 22:59:55.886525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.886444    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd" event={"ID":"e0dd0998-0241-4637-b452-e3146fd29b32","Type":"ContainerDied","Data":"3d05d09acb885129076a6b1bf0426c35b8361cd8752867dde7e9197d53735170"}
Oct 29 22:59:55.886525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.886457    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d05d09acb885129076a6b1bf0426c35b8361cd8752867dde7e9197d53735170"
Oct 29 22:59:55.886525 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.886490    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:59:55.915117 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.915085    4451 scope.go:122] "RemoveContainer" containerID="db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1"
Oct 29 22:59:55.921279 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:55.921252    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1\": container with ID starting with db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1 not found: ID does not exist" containerID="db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1"
Oct 29 22:59:55.921418 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.921285    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1"} err="failed to get container status \"db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1\": rpc error: code = NotFound desc = could not find container \"db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1\": container with ID starting with db14a155cababf47e1de07162bf21936a9336bd67e799dbe554944416c5717c1 not found: ID does not exist"
Oct 29 22:59:55.966650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.966555    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm\") pod \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") "
Oct 29 22:59:55.966650 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.966612    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache\") pod \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") "
Oct 29 22:59:55.966875 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.966701    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm\") pod \"e0dd0998-0241-4637-b452-e3146fd29b32\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") "
Oct 29 22:59:55.966875 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.966813    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir\") pod \"e0dd0998-0241-4637-b452-e3146fd29b32\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") "
Oct 29 22:59:55.967061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.966897    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home\") pod \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") "
Oct 29 22:59:55.967061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.966972    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir\") pod \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") "
Oct 29 22:59:55.967061 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967017    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home\") pod \"e0dd0998-0241-4637-b452-e3146fd29b32\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") "
Oct 29 22:59:55.967218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967068    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs\") pod \"e0dd0998-0241-4637-b452-e3146fd29b32\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") "
Oct 29 22:59:55.967218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967107    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc\" (UniqueName: \"kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc\") pod \"e0dd0998-0241-4637-b452-e3146fd29b32\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") "
Oct 29 22:59:55.967218 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967194    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs\") pod \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") "
Oct 29 22:59:55.967376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967232    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache\") pod \"e0dd0998-0241-4637-b452-e3146fd29b32\" (UID: \"e0dd0998-0241-4637-b452-e3146fd29b32\") "
Oct 29 22:59:55.967376 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967303    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home" pod "bb31020f-0dc3-40f9-9f52-72462ba65bdc" (UID: "bb31020f-0dc3-40f9-9f52-72462ba65bdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.967477 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967393    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt\" (UniqueName: \"kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt\") pod \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\" (UID: \"bb31020f-0dc3-40f9-9f52-72462ba65bdc\") "
Oct 29 22:59:55.967601 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967577    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache" pod "e0dd0998-0241-4637-b452-e3146fd29b32" (UID: "e0dd0998-0241-4637-b452-e3146fd29b32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.967799 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.967778    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir" pod "bb31020f-0dc3-40f9-9f52-72462ba65bdc" (UID: "bb31020f-0dc3-40f9-9f52-72462ba65bdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.968049 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968027    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home" pod "e0dd0998-0241-4637-b452-e3146fd29b32" (UID: "e0dd0998-0241-4637-b452-e3146fd29b32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.968406 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968378    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache" pod "bb31020f-0dc3-40f9-9f52-72462ba65bdc" (UID: "bb31020f-0dc3-40f9-9f52-72462ba65bdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.968637 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968617    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.968747 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968734    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.968841 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968830    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.969279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968925    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.969279 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.968174    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir" pod "e0dd0998-0241-4637-b452-e3146fd29b32" (UID: "e0dd0998-0241-4637-b452-e3146fd29b32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.970039 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.970005    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm" pod "bb31020f-0dc3-40f9-9f52-72462ba65bdc" (UID: "bb31020f-0dc3-40f9-9f52-72462ba65bdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.971457 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.971430    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs" pod "e0dd0998-0241-4637-b452-e3146fd29b32" (UID: "e0dd0998-0241-4637-b452-e3146fd29b32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:55.973367 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.973327    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt" pod "bb31020f-0dc3-40f9-9f52-72462ba65bdc" (UID: "bb31020f-0dc3-40f9-9f52-72462ba65bdc"). InnerVolumeSpecName "kube-api-access-hdxkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:55.973496 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.973477    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs" pod "bb31020f-0dc3-40f9-9f52-72462ba65bdc" (UID: "bb31020f-0dc3-40f9-9f52-72462ba65bdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:55.974172 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.974147    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm" pod "e0dd0998-0241-4637-b452-e3146fd29b32" (UID: "e0dd0998-0241-4637-b452-e3146fd29b32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.975322 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:55.975296    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc" pod "e0dd0998-0241-4637-b452-e3146fd29b32" (UID: "e0dd0998-0241-4637-b452-e3146fd29b32"). InnerVolumeSpecName "kube-api-access-v4fxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:56.070560 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070528    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb31020f-0dc3-40f9-9f52-72462ba65bdc-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070560 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070559    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxkt\" (UniqueName: \"kubernetes.io/projected/bb31020f-0dc3-40f9-9f52-72462ba65bdc-kube-api-access-hdxkt\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070569    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070578    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb31020f-0dc3-40f9-9f52-72462ba65bdc-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070586    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070594    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0dd0998-0241-4637-b452-e3146fd29b32-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070601    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd0998-0241-4637-b452-e3146fd29b32-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.070759 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.070609    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fxc\" (UniqueName: \"kubernetes.io/projected/e0dd0998-0241-4637-b452-e3146fd29b32-kube-api-access-v4fxc\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.219588 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.219509    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"]
Oct 29 22:59:56.223151 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.223123    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb"]
Oct 29 22:59:56.820932 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.820880    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/distinct-w6-784e7a-kserve-88dffc699-w59mb" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:59:56.896438 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.896412    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"
Oct 29 22:59:56.942093 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.942053    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"]
Oct 29 22:59:56.951440 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:56.951414    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/distinct-2-w6-784e7a-kserve-588c9f7bd4-sklqd"]
Oct 29 22:59:58.165724 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:58.165687    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" path="/var/lib/kubelet/pods/bb31020f-0dc3-40f9-9f52-72462ba65bdc/volumes"
Oct 29 22:59:58.166191 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:58.166175    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0dd0998-0241-4637-b452-e3146fd29b32" path="/var/lib/kubelet/pods/e0dd0998-0241-4637-b452-e3146fd29b32/volumes"
Oct 29 22:59:58.925180 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:58.925146    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"]
Oct 29 22:59:58.925400 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:58.925374    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6" podUID="1fc86164-b7a8-447d-8656-dc6016c4a134" containerName="maas-api" containerID="cri-o://99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999" gracePeriod=30
Oct 29 22:59:59.523941 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.516965    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:59:59.587944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.586044    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls\") pod \"1fc86164-b7a8-447d-8656-dc6016c4a134\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") "
Oct 29 22:59:59.587944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.586234    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs\") pod \"1fc86164-b7a8-447d-8656-dc6016c4a134\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") "
Oct 29 22:59:59.587944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.586311    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2\" (UniqueName: \"kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2\") pod \"1fc86164-b7a8-447d-8656-dc6016c4a134\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") "
Oct 29 22:59:59.587944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.586394    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca\") pod \"1fc86164-b7a8-447d-8656-dc6016c4a134\" (UID: \"1fc86164-b7a8-447d-8656-dc6016c4a134\") "
Oct 29 22:59:59.587944 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.587891    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca" pod "1fc86164-b7a8-447d-8656-dc6016c4a134" (UID: "1fc86164-b7a8-447d-8656-dc6016c4a134"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:59.602982 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.591460    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls" pod "1fc86164-b7a8-447d-8656-dc6016c4a134" (UID: "1fc86164-b7a8-447d-8656-dc6016c4a134"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:59.614800 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.612229    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2" pod "1fc86164-b7a8-447d-8656-dc6016c4a134" (UID: "1fc86164-b7a8-447d-8656-dc6016c4a134"). InnerVolumeSpecName "kube-api-access-r5mp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:59.616610 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.616575    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs" pod "1fc86164-b7a8-447d-8656-dc6016c4a134" (UID: "1fc86164-b7a8-447d-8656-dc6016c4a134"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:59.694600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.694342    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:59.694600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.694388    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mp2\" (UniqueName: \"kubernetes.io/projected/1fc86164-b7a8-447d-8656-dc6016c4a134-kube-api-access-r5mp2\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:59.694600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.694406    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc86164-b7a8-447d-8656-dc6016c4a134-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:59.694600 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.694420    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc86164-b7a8-447d-8656-dc6016c4a134-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 22:59:59.918718 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.918687    4451 generic.go:356] "Generic (PLEG): container finished" podID="1fc86164-b7a8-447d-8656-dc6016c4a134" containerID="99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999" exitCode=0
Oct 29 22:59:59.918938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.918757    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"
Oct 29 22:59:59.918938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.918774    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6" event={"ID":"1fc86164-b7a8-447d-8656-dc6016c4a134","Type":"ContainerDied","Data":"99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999"}
Oct 29 22:59:59.918938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.918869    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6" event={"ID":"1fc86164-b7a8-447d-8656-dc6016c4a134","Type":"ContainerDied","Data":"fb25fb0c5ab17b5461dbae23c7acbf75b1ea3399a61709721ef3722dd5578dda"}
Oct 29 22:59:59.918938 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.918899    4451 scope.go:122] "RemoveContainer" containerID="99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999"
Oct 29 22:59:59.930130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.929759    4451 scope.go:122] "RemoveContainer" containerID="99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999"
Oct 29 22:59:59.930130 ip-10-0-143-15 kubenswrapper[4451]: E1029 22:59:59.930056    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999\": container with ID starting with 99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999 not found: ID does not exist" containerID="99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999"
Oct 29 22:59:59.930130 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.930084    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999"} err="failed to get container status \"99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999\": rpc error: code = NotFound desc = could not find container \"99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999\": container with ID starting with 99d00e5a035cde6f98e808c861d4c9b5361e2acb77364e4205611156d1a19999 not found: ID does not exist"
Oct 29 22:59:59.964740 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.964680    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"]
Oct 29 22:59:59.974303 ip-10-0-143-15 kubenswrapper[4451]: I1029 22:59:59.974212    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f8ca96-7769dff8d-dxpk6"]
Oct 29 23:00:00.168832 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:00.168796    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc86164-b7a8-447d-8656-dc6016c4a134" path="/var/lib/kubelet/pods/1fc86164-b7a8-447d-8656-dc6016c4a134/volumes"
Oct 29 23:00:07.567837 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:07.566175    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"]
Oct 29 23:00:07.567837 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:07.566435    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy" containerID="cri-o://67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5" gracePeriod=30
Oct 29 23:00:07.575860 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:07.575799    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"]
Oct 29 23:00:07.577065 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:07.576487    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy" containerID="cri-o://7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e" gracePeriod=30
Oct 29 23:00:07.582837 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:07.582742    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"]
Oct 29 23:00:07.583248 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:07.583222    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy" containerID="cri-o://c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d" gracePeriod=30
Oct 29 23:00:08.275850 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.275818    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"]
Oct 29 23:00:08.276937 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.276893    4451 cpu_manager.go:414] "containerMap: removing container" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main"
Oct 29 23:00:08.276937 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.276936    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main"
Oct 29 23:00:08.277246 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.276957    4451 cpu_manager.go:414] "containerMap: removing container" podUID="1fc86164-b7a8-447d-8656-dc6016c4a134" containerName="maas-api"
Oct 29 23:00:08.277246 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.276972    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc86164-b7a8-447d-8656-dc6016c4a134" containerName="maas-api"
Oct 29 23:00:08.277246 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.277020    4451 cpu_manager.go:414] "containerMap: removing container" podUID="e0dd0998-0241-4637-b452-e3146fd29b32" containerName="main"
Oct 29 23:00:08.277246 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.277028    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0dd0998-0241-4637-b452-e3146fd29b32" containerName="main"
Oct 29 23:00:08.278801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.277238    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0dd0998-0241-4637-b452-e3146fd29b32" containerName="main"
Oct 29 23:00:08.278801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.277289    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb31020f-0dc3-40f9-9f52-72462ba65bdc" containerName="main"
Oct 29 23:00:08.278801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.277301    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc86164-b7a8-447d-8656-dc6016c4a134" containerName="maas-api"
Oct 29 23:00:08.283019 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.283002    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.295752 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.295729    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:08.305438 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.305408    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:08.307390 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.307363    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"]
Oct 29 23:00:08.462846 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.462806    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.463071 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.463018    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcb6n\" (UniqueName: \"kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.463155 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.463088    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.565133 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.565052    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.565283 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.565199    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.565333 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.565321    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcb6n\" (UniqueName: \"kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.565824 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.565790    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.567622 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.567595    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.583434 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.583410    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcb6n\" (UniqueName: \"kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n\") pod \"maas-api-revoke-sub-f6b23552-nkh9f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.599291 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.599266    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:08.905678 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.905642    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"]
Oct 29 23:00:08.905861 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:00:08.905837    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9792b127_6a0c_4287_ab5d_70bcd54a211f.slice/crio-c3d2448c695d8f4ff72365f8cd7498dcb5086b2a77f415efbf594a17389de67d WatchSource:0}: Error finding container c3d2448c695d8f4ff72365f8cd7498dcb5086b2a77f415efbf594a17389de67d: Status 404 returned error can't find the container with id c3d2448c695d8f4ff72365f8cd7498dcb5086b2a77f415efbf594a17389de67d
Oct 29 23:00:08.992394 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:08.992358    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f" event={"ID":"9792b127-6a0c-4287-ab5d-70bcd54a211f","Type":"ContainerStarted","Data":"c3d2448c695d8f4ff72365f8cd7498dcb5086b2a77f415efbf594a17389de67d"}
Oct 29 23:00:10.001203 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:10.001160    4451 generic.go:356] "Generic (PLEG): container finished" podID="9792b127-6a0c-4287-ab5d-70bcd54a211f" containerID="12b08a423e72c0c9a6b05891f4ef53819abf95291d28b2c024bab15bb7959ea7" exitCode=0
Oct 29 23:00:10.001599 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:10.001240    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f" event={"ID":"9792b127-6a0c-4287-ab5d-70bcd54a211f","Type":"ContainerDied","Data":"12b08a423e72c0c9a6b05891f4ef53819abf95291d28b2c024bab15bb7959ea7"}
Oct 29 23:00:11.212947 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.212884    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:11.319530 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.319506    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca\") pod \"9792b127-6a0c-4287-ab5d-70bcd54a211f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") "
Oct 29 23:00:11.319703 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.319618    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca\") pod \"9792b127-6a0c-4287-ab5d-70bcd54a211f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") "
Oct 29 23:00:11.319820 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.319804    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n\" (UniqueName: \"kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n\") pod \"9792b127-6a0c-4287-ab5d-70bcd54a211f\" (UID: \"9792b127-6a0c-4287-ab5d-70bcd54a211f\") "
Oct 29 23:00:11.324710 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.324200    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n" pod "9792b127-6a0c-4287-ab5d-70bcd54a211f" (UID: "9792b127-6a0c-4287-ab5d-70bcd54a211f"). InnerVolumeSpecName "kube-api-access-xcb6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:11.324710 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.324533    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca" pod "9792b127-6a0c-4287-ab5d-70bcd54a211f" (UID: "9792b127-6a0c-4287-ab5d-70bcd54a211f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:11.327209 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.327185    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca" pod "9792b127-6a0c-4287-ab5d-70bcd54a211f" (UID: "9792b127-6a0c-4287-ab5d-70bcd54a211f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:11.421930 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.421884    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcb6n\" (UniqueName: \"kubernetes.io/projected/9792b127-6a0c-4287-ab5d-70bcd54a211f-kube-api-access-xcb6n\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.422086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.421942    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-xks-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.422086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:11.421960    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9792b127-6a0c-4287-ab5d-70bcd54a211f-maas-api-openshift-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.015426 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:12.015389    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"
Oct 29 23:00:12.015426 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:12.015405    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f" event={"ID":"9792b127-6a0c-4287-ab5d-70bcd54a211f","Type":"ContainerDied","Data":"c3d2448c695d8f4ff72365f8cd7498dcb5086b2a77f415efbf594a17389de67d"}
Oct 29 23:00:12.015638 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:12.015448    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3d2448c695d8f4ff72365f8cd7498dcb5086b2a77f415efbf594a17389de67d"
Oct 29 23:00:12.942964 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:12.936443    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 23:00:12.964986 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:12.964953    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 23:00:12.977183 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:12.977127    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 23:00:13.066286 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.066239    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.066536 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.066411    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.066630 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.066576    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.066726 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.066705    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.066901 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.066882    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.067172 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.067152    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.069725 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.069685    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.070387 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.070357    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.070827 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.070801    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.072555 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.072515    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.073038 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.073012    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.073205 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.073043    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a8d410-df76-4882-9764-3227df225944-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.073205 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.073055    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.082956 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.081171    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.090410 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.090364    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.091725 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.091702    4451 generic.go:356] "Generic (PLEG): container finished" podID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerID="7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e" exitCode=0
Oct 29 23:00:13.092245 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.092220    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" event={"ID":"37ccc1a0-98cb-4366-bb9b-6bca93a999a1","Type":"ContainerDied","Data":"7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e"}
Oct 29 23:00:13.092245 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.091902    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"
Oct 29 23:00:13.092433 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.092345    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s" event={"ID":"37ccc1a0-98cb-4366-bb9b-6bca93a999a1","Type":"ContainerDied","Data":"fc0883e091f9cd7e933d863f3a4ca0594ff9f238aab125f73bbc5a488592036c"}
Oct 29 23:00:13.092433 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.092393    4451 scope.go:122] "RemoveContainer" containerID="7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e"
Oct 29 23:00:13.095055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.094771    4451 generic.go:356] "Generic (PLEG): container finished" podID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerID="67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5" exitCode=0
Oct 29 23:00:13.095055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.094853    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"
Oct 29 23:00:13.095055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.094854    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" event={"ID":"2b6a29c6-375d-40d5-a2f2-144bc504e51f","Type":"ContainerDied","Data":"67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5"}
Oct 29 23:00:13.095055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.094970    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl" event={"ID":"2b6a29c6-375d-40d5-a2f2-144bc504e51f","Type":"ContainerDied","Data":"073d7b2a5107fe6ac95960e537e03eee717feaac2774dd108a193e583ffc2cc7"}
Oct 29 23:00:13.097394 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.097367    4451 generic.go:356] "Generic (PLEG): container finished" podID="e9a8d410-df76-4882-9764-3227df225944" containerID="c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d" exitCode=0
Oct 29 23:00:13.097492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.097419    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" event={"ID":"e9a8d410-df76-4882-9764-3227df225944","Type":"ContainerDied","Data":"c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d"}
Oct 29 23:00:13.097492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.097456    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd" event={"ID":"e9a8d410-df76-4882-9764-3227df225944","Type":"ContainerDied","Data":"85f2822b2316d1c2edf17a59d48635818ac6e6f96085bf1342fbfe3b53486db2"}
Oct 29 23:00:13.097617 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.097533    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"
Oct 29 23:00:13.109459 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.109436    4451 scope.go:122] "RemoveContainer" containerID="7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e"
Oct 29 23:00:13.109788 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:13.109762    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e\": container with ID starting with 7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e not found: ID does not exist" containerID="7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e"
Oct 29 23:00:13.109889 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.109798    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e"} err="failed to get container status \"7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e\": rpc error: code = NotFound desc = could not find container \"7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e\": container with ID starting with 7e34a10df097dc6cea67a0b8e61ff3d1d0144d6cfdcbfe15dfa2b5239454be6e not found: ID does not exist"
Oct 29 23:00:13.109889 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.109820    4451 scope.go:122] "RemoveContainer" containerID="67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5"
Oct 29 23:00:13.135439 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.133510    4451 scope.go:122] "RemoveContainer" containerID="67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5"
Oct 29 23:00:13.135641 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:13.135614    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5\": container with ID starting with 67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5 not found: ID does not exist" containerID="67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5"
Oct 29 23:00:13.135716 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.135650    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5"} err="failed to get container status \"67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5\": rpc error: code = NotFound desc = could not find container \"67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5\": container with ID starting with 67dffbd1bcd878b2e514add66cfaae7514ac6fa7ce6dc5de15cdf4876e1b2ef5 not found: ID does not exist"
Oct 29 23:00:13.135716 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.135668    4451 scope.go:122] "RemoveContainer" containerID="c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d"
Oct 29 23:00:13.147095 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.147070    4451 scope.go:122] "RemoveContainer" containerID="c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d"
Oct 29 23:00:13.147484 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:13.147455    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d\": container with ID starting with c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d not found: ID does not exist" containerID="c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d"
Oct 29 23:00:13.147590 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.147493    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d"} err="failed to get container status \"c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d\": rpc error: code = NotFound desc = could not find container \"c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d\": container with ID starting with c8a61f855789c7e952e7aa6759d01f41bb7660d82e104b2ef68f6c34bf6f259d not found: ID does not exist"
Oct 29 23:00:13.174087 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174067    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.174185 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174104    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174185 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174134    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.174185 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174158    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174308 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174186    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.174308 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174232    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.174308 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174276    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.174460 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174318    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.174460 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174351    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174460 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174393    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.174460 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174447    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174476    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174504    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174537    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174571    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174572    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174646    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174684    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174718    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174759    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket\") pod \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\" (UID: \"2b6a29c6-375d-40d5-a2f2-144bc504e51f\") "
Oct 29 23:00:13.174795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174794    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket\") pod \"e9a8d410-df76-4882-9764-3227df225944\" (UID: \"e9a8d410-df76-4882-9764-3227df225944\") "
Oct 29 23:00:13.175417 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174843    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket\") pod \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\" (UID: \"37ccc1a0-98cb-4366-bb9b-6bca93a999a1\") "
Oct 29 23:00:13.175417 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174646    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.175417 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.174786    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.175417 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175036    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.175417 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175165    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.175417 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175364    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.175737 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175700    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.175799 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175738    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.175799 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175763    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.175799 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175778    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.175799 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175793    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.176085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175806    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.176085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175823    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.176085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175836    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.176085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175849    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.176085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.175905    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.176382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.176083    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.176382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.176144    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.176382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.176283    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.176382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.176360    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.177413 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.177392    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "kube-api-access-jqc5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.177524 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.177495    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "kube-api-access-vmsrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.177608 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.177452    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "kube-api-access-5jrdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.177608 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.177509    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.177802 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.177779    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.178476 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.178455    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy" pod "37ccc1a0-98cb-4366-bb9b-6bca93a999a1" (UID: "37ccc1a0-98cb-4366-bb9b-6bca93a999a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.178789 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.178771    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.178851 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.178838    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.178975 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.178961    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo" pod "2b6a29c6-375d-40d5-a2f2-144bc504e51f" (UID: "2b6a29c6-375d-40d5-a2f2-144bc504e51f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.179264 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.179252    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy" pod "e9a8d410-df76-4882-9764-3227df225944" (UID: "e9a8d410-df76-4882-9764-3227df225944"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.277643 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277605    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.277643 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277634    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jrdd\" (UniqueName: \"kubernetes.io/projected/e9a8d410-df76-4882-9764-3227df225944-kube-api-access-5jrdd\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.277643 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277645    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.277643 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277653    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqc5f\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-kube-api-access-jqc5f\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277661    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277674    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277685    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277693    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277700    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277708    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a8d410-df76-4882-9764-3227df225944-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277717    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277724    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmsrk\" (UniqueName: \"kubernetes.io/projected/2b6a29c6-375d-40d5-a2f2-144bc504e51f-kube-api-access-vmsrk\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277732    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6a29c6-375d-40d5-a2f2-144bc504e51f-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277741    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a8d410-df76-4882-9764-3227df225944-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.278015 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.277754    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ccc1a0-98cb-4366-bb9b-6bca93a999a1-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.438231 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.438205    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"]
Oct 29 23:00:13.442374 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.442350    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mvw5s"]
Oct 29 23:00:13.472092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.472068    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"]
Oct 29 23:00:13.478444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.478422    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-glsxl"]
Oct 29 23:00:13.507782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.507756    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"]
Oct 29 23:00:13.513327 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:13.513306    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-rt6nd"]
Oct 29 23:00:14.163867 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:14.163836    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" path="/var/lib/kubelet/pods/2b6a29c6-375d-40d5-a2f2-144bc504e51f/volumes"
Oct 29 23:00:14.164429 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:14.164410    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" path="/var/lib/kubelet/pods/37ccc1a0-98cb-4366-bb9b-6bca93a999a1/volumes"
Oct 29 23:00:14.164876 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:14.164862    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a8d410-df76-4882-9764-3227df225944" path="/var/lib/kubelet/pods/e9a8d410-df76-4882-9764-3227df225944/volumes"
Oct 29 23:00:18.751053 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:18.750985    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:00:21.297575 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:21.297542    4451 scope.go:122] "RemoveContainer" containerID="f7537e08e4c0a7d1d4c31b5c507ef7c76bd4dd606091ecd58e043a5e5e1a55c0"
Oct 29 23:00:33.220665 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.220633    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"]
Oct 29 23:00:33.221191 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.220890    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" podUID="d111483e-be3a-4478-bbfa-c55215de5da4" containerName="maas-api" containerID="cri-o://880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6" gracePeriod=30
Oct 29 23:00:33.234251 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.233401    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" podUID="d111483e-be3a-4478-bbfa-c55215de5da4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.77:8443/health\": dial tcp 10.133.0.77:8443: connect: connection refused"
Oct 29 23:00:33.623206 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.623151    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 23:00:33.812405 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.812357    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf\" (UniqueName: \"kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf\") pod \"d111483e-be3a-4478-bbfa-c55215de5da4\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") "
Oct 29 23:00:33.812787 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.812759    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs\") pod \"d111483e-be3a-4478-bbfa-c55215de5da4\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") "
Oct 29 23:00:33.813198 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.813150    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls\") pod \"d111483e-be3a-4478-bbfa-c55215de5da4\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") "
Oct 29 23:00:33.813370 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.813351    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca\") pod \"d111483e-be3a-4478-bbfa-c55215de5da4\" (UID: \"d111483e-be3a-4478-bbfa-c55215de5da4\") "
Oct 29 23:00:33.814795 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.814771    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca" pod "d111483e-be3a-4478-bbfa-c55215de5da4" (UID: "d111483e-be3a-4478-bbfa-c55215de5da4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:33.821804 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.821767    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf" pod "d111483e-be3a-4478-bbfa-c55215de5da4" (UID: "d111483e-be3a-4478-bbfa-c55215de5da4"). InnerVolumeSpecName "kube-api-access-h9gbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:33.822208 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.822177    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs" pod "d111483e-be3a-4478-bbfa-c55215de5da4" (UID: "d111483e-be3a-4478-bbfa-c55215de5da4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:33.824189 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.824168    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls" pod "d111483e-be3a-4478-bbfa-c55215de5da4" (UID: "d111483e-be3a-4478-bbfa-c55215de5da4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:33.915929 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.915888    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9gbf\" (UniqueName: \"kubernetes.io/projected/d111483e-be3a-4478-bbfa-c55215de5da4-kube-api-access-h9gbf\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:33.916086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.915960    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:33.916086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.915971    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d111483e-be3a-4478-bbfa-c55215de5da4-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:33.916086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:33.915979    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d111483e-be3a-4478-bbfa-c55215de5da4-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:34.283261 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.283230    4451 generic.go:356] "Generic (PLEG): container finished" podID="d111483e-be3a-4478-bbfa-c55215de5da4" containerID="880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6" exitCode=0
Oct 29 23:00:34.283656 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.283274    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"
Oct 29 23:00:34.283656 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.283309    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" event={"ID":"d111483e-be3a-4478-bbfa-c55215de5da4","Type":"ContainerDied","Data":"880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6"}
Oct 29 23:00:34.283656 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.283363    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds" event={"ID":"d111483e-be3a-4478-bbfa-c55215de5da4","Type":"ContainerDied","Data":"ed6b84325e74f64bf75ac1138b22b839855b067190021b558f2c55b20705f425"}
Oct 29 23:00:34.283656 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.283385    4451 scope.go:122] "RemoveContainer" containerID="880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6"
Oct 29 23:00:34.294099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.294078    4451 scope.go:122] "RemoveContainer" containerID="880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6"
Oct 29 23:00:34.294358 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:34.294338    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6\": container with ID starting with 880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6 not found: ID does not exist" containerID="880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6"
Oct 29 23:00:34.294441 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.294366    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6"} err="failed to get container status \"880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6\": rpc error: code = NotFound desc = could not find container \"880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6\": container with ID starting with 880cb5ea413e892fb162f5d5be366bc5dafa9fb0998e32207f623124b7ae42d6 not found: ID does not exist"
Oct 29 23:00:34.327256 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.327232    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"]
Oct 29 23:00:34.336328 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:34.336306    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3bd0f6-55f88699d-vrnds"]
Oct 29 23:00:36.166699 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:36.166663    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d111483e-be3a-4478-bbfa-c55215de5da4" path="/var/lib/kubelet/pods/d111483e-be3a-4478-bbfa-c55215de5da4/volumes"
Oct 29 23:00:41.081708 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:41.081653    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"]
Oct 29 23:00:41.082790 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:41.081991    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy" containerID="cri-o://03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5" gracePeriod=30
Oct 29 23:00:46.358966 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.358878    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 23:00:46.385987 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.385958    4451 generic.go:356] "Generic (PLEG): container finished" podID="5130665c-f681-4734-9356-0046af52ee3c" containerID="03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5" exitCode=0
Oct 29 23:00:46.386092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.386000    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"
Oct 29 23:00:46.386092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.386037    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" event={"ID":"5130665c-f681-4734-9356-0046af52ee3c","Type":"ContainerDied","Data":"03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5"}
Oct 29 23:00:46.386092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.386088    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc" event={"ID":"5130665c-f681-4734-9356-0046af52ee3c","Type":"ContainerDied","Data":"a2f29e325113f9f60a4e4dcf6d66a526f08b24031ab73c7ad6722a66b01bfae3"}
Oct 29 23:00:46.386203 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.386110    4451 scope.go:122] "RemoveContainer" containerID="03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5"
Oct 29 23:00:46.396338 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396319    4451 scope.go:122] "RemoveContainer" containerID="03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5"
Oct 29 23:00:46.396408 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396377    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396514 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396497    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396573 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396538    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396636 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:46.396579    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5\": container with ID starting with 03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5 not found: ID does not exist" containerID="03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5"
Oct 29 23:00:46.396636 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396612    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5"} err="failed to get container status \"03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5\": rpc error: code = NotFound desc = could not find container \"03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5\": container with ID starting with 03caa44faae5a0a388d7603f3c6451320f76d8e63e863aa1c1960707f27649a5 not found: ID does not exist"
Oct 29 23:00:46.396723 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396650    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396723 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396683    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.396723 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396717    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396885 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396765    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396885 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396797    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.396885 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396821    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.396885 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396833    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.396885 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.396878    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.397261 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397002    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket\") pod \"5130665c-f681-4734-9356-0046af52ee3c\" (UID: \"5130665c-f681-4734-9356-0046af52ee3c\") "
Oct 29 23:00:46.397261 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397230    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.397373 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397254    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:46.397639 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397618    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.397764 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397747    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.397764 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397763    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5130665c-f681-4734-9356-0046af52ee3c-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.397956 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397774    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.397956 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.397782    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.398900 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.398882    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:46.399118 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.399096    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.399382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.399366    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "kube-api-access-4msdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.399430 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.399414    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token" pod "5130665c-f681-4734-9356-0046af52ee3c" (UID: "5130665c-f681-4734-9356-0046af52ee3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.499679 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.499645    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.499679 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.499673    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5130665c-f681-4734-9356-0046af52ee3c-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.499679 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.499684    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4msdz\" (UniqueName: \"kubernetes.io/projected/5130665c-f681-4734-9356-0046af52ee3c-kube-api-access-4msdz\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.499941 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.499693    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5130665c-f681-4734-9356-0046af52ee3c-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.524197 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.524163    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"]
Oct 29 23:00:46.731365 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.731312    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"]
Oct 29 23:00:46.734964 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:46.734932    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-shmwc"]
Oct 29 23:00:48.165533 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:48.165493    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5130665c-f681-4734-9356-0046af52ee3c" path="/var/lib/kubelet/pods/5130665c-f681-4734-9356-0046af52ee3c/volumes"
Oct 29 23:00:48.742639 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:48.742597    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:00:50.303941 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.303891    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"]
Oct 29 23:00:50.304652 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.304618    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd" podUID="193bef37-cee5-4696-967d-db3ad0bd76c8" containerName="maas-api" containerID="cri-o://46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2" gracePeriod=30
Oct 29 23:00:50.947329 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.947297    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 23:00:50.979360 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.979319    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca\") pod \"193bef37-cee5-4696-967d-db3ad0bd76c8\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") "
Oct 29 23:00:50.979535 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.979487    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29\" (UniqueName: \"kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29\") pod \"193bef37-cee5-4696-967d-db3ad0bd76c8\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") "
Oct 29 23:00:50.979582 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.979535    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs\") pod \"193bef37-cee5-4696-967d-db3ad0bd76c8\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") "
Oct 29 23:00:50.979641 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.979625    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls\") pod \"193bef37-cee5-4696-967d-db3ad0bd76c8\" (UID: \"193bef37-cee5-4696-967d-db3ad0bd76c8\") "
Oct 29 23:00:50.979848 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.979819    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca" pod "193bef37-cee5-4696-967d-db3ad0bd76c8" (UID: "193bef37-cee5-4696-967d-db3ad0bd76c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:50.983379 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.983354    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29" pod "193bef37-cee5-4696-967d-db3ad0bd76c8" (UID: "193bef37-cee5-4696-967d-db3ad0bd76c8"). InnerVolumeSpecName "kube-api-access-pzl29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:50.983726 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.983465    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193bef37-cee5-4696-967d-db3ad0bd76c8-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:50.983726 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.983610    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs" pod "193bef37-cee5-4696-967d-db3ad0bd76c8" (UID: "193bef37-cee5-4696-967d-db3ad0bd76c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:50.984099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:50.984073    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls" pod "193bef37-cee5-4696-967d-db3ad0bd76c8" (UID: "193bef37-cee5-4696-967d-db3ad0bd76c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:51.085646 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.085607    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzl29\" (UniqueName: \"kubernetes.io/projected/193bef37-cee5-4696-967d-db3ad0bd76c8-kube-api-access-pzl29\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:51.085646 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.085647    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:51.085843 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.085662    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193bef37-cee5-4696-967d-db3ad0bd76c8-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:00:51.420574 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.420537    4451 generic.go:356] "Generic (PLEG): container finished" podID="193bef37-cee5-4696-967d-db3ad0bd76c8" containerID="46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2" exitCode=0
Oct 29 23:00:51.421049 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.420584    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"
Oct 29 23:00:51.421049 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.420610    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd" event={"ID":"193bef37-cee5-4696-967d-db3ad0bd76c8","Type":"ContainerDied","Data":"46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2"}
Oct 29 23:00:51.421049 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.420685    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd" event={"ID":"193bef37-cee5-4696-967d-db3ad0bd76c8","Type":"ContainerDied","Data":"fb4f1106e4a61c6af78dde71ed3fec064369f5ec6f46303db8a498d5606d9fbe"}
Oct 29 23:00:51.421049 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.420718    4451 scope.go:122] "RemoveContainer" containerID="46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2"
Oct 29 23:00:51.433175 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.433152    4451 scope.go:122] "RemoveContainer" containerID="46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2"
Oct 29 23:00:51.433472 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:00:51.433452    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2\": container with ID starting with 46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2 not found: ID does not exist" containerID="46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2"
Oct 29 23:00:51.433547 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.433478    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2"} err="failed to get container status \"46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2\": rpc error: code = NotFound desc = could not find container \"46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2\": container with ID starting with 46f16d2824506904bafff270a11c53510d7e518969db8edd98ebe4757f1204d2 not found: ID does not exist"
Oct 29 23:00:51.481008 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.480977    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"]
Oct 29 23:00:51.489193 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:51.489137    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-784e7a-7d4687f97d-68tkd"]
Oct 29 23:00:52.167636 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:52.167600    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193bef37-cee5-4696-967d-db3ad0bd76c8" path="/var/lib/kubelet/pods/193bef37-cee5-4696-967d-db3ad0bd76c8/volumes"
Oct 29 23:00:58.136766 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:58.136732    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"]
Oct 29 23:00:58.137264 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:00:58.137024    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" containerID="cri-o://d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387" gracePeriod=30
Oct 29 23:01:01.547769 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.547717    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main" containerID="cri-o://afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6" gracePeriod=45
Oct 29 23:01:01.818835 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.818805    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 23:01:01.874107 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.874072    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache\") pod \"de5c8db1-3048-414d-822b-2aa15c83932b\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") "
Oct 29 23:01:01.874372 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.874352    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home\") pod \"de5c8db1-3048-414d-822b-2aa15c83932b\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") "
Oct 29 23:01:01.874561 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.874512    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir\") pod \"de5c8db1-3048-414d-822b-2aa15c83932b\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") "
Oct 29 23:01:01.874561 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.874548    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2\" (UniqueName: \"kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2\") pod \"de5c8db1-3048-414d-822b-2aa15c83932b\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") "
Oct 29 23:01:01.874701 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.874634    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm\") pod \"de5c8db1-3048-414d-822b-2aa15c83932b\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") "
Oct 29 23:01:01.874701 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.874677    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs\") pod \"de5c8db1-3048-414d-822b-2aa15c83932b\" (UID: \"de5c8db1-3048-414d-822b-2aa15c83932b\") "
Oct 29 23:01:01.875775 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.875749    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir" pod "de5c8db1-3048-414d-822b-2aa15c83932b" (UID: "de5c8db1-3048-414d-822b-2aa15c83932b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.876009 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.875986    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache" pod "de5c8db1-3048-414d-822b-2aa15c83932b" (UID: "de5c8db1-3048-414d-822b-2aa15c83932b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.876272 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.876228    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home" pod "de5c8db1-3048-414d-822b-2aa15c83932b" (UID: "de5c8db1-3048-414d-822b-2aa15c83932b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.878988 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.878907    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm" pod "de5c8db1-3048-414d-822b-2aa15c83932b" (UID: "de5c8db1-3048-414d-822b-2aa15c83932b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.879274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.879250    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs" pod "de5c8db1-3048-414d-822b-2aa15c83932b" (UID: "de5c8db1-3048-414d-822b-2aa15c83932b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:01.889104 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.889077    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2" pod "de5c8db1-3048-414d-822b-2aa15c83932b" (UID: "de5c8db1-3048-414d-822b-2aa15c83932b"). InnerVolumeSpecName "kube-api-access-7ngf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:01.977241 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.977204    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.977430 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.977264    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.977430 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.977280    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ngf2\" (UniqueName: \"kubernetes.io/projected/de5c8db1-3048-414d-822b-2aa15c83932b-kube-api-access-7ngf2\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.977430 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.977296    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.977430 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.977309    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5c8db1-3048-414d-822b-2aa15c83932b-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.977430 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:01.977322    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5c8db1-3048-414d-822b-2aa15c83932b-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.485871 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.485786    4451 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:02.485871 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.485857    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:02.551943 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.551898    4451 generic.go:356] "Generic (PLEG): container finished" podID="de5c8db1-3048-414d-822b-2aa15c83932b" containerID="afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6" exitCode=0
Oct 29 23:01:02.552414 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.551971    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" event={"ID":"de5c8db1-3048-414d-822b-2aa15c83932b","Type":"ContainerDied","Data":"afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6"}
Oct 29 23:01:02.552414 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.551998    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"
Oct 29 23:01:02.552414 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.552042    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48" event={"ID":"de5c8db1-3048-414d-822b-2aa15c83932b","Type":"ContainerDied","Data":"2585c3af9a584d8f6c897873f80d0658038f2443bd4cf97611fbce043b4fa3ed"}
Oct 29 23:01:02.552414 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.552055    4451 scope.go:122] "RemoveContainer" containerID="afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6"
Oct 29 23:01:02.563635 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.563616    4451 scope.go:122] "RemoveContainer" containerID="afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6"
Oct 29 23:01:02.563907 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:01:02.563886    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6\": container with ID starting with afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6 not found: ID does not exist" containerID="afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6"
Oct 29 23:01:02.564004 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.563932    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6"} err="failed to get container status \"afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6\": rpc error: code = NotFound desc = could not find container \"afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6\": container with ID starting with afa9b59c2b6a235d5ddd3635d3404b0bd163899390e20aa4ebe57a455f1e5fd6 not found: ID does not exist"
Oct 29 23:01:02.620087 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.620055    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"]
Oct 29 23:01:02.629639 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:02.629614    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-w3-43b37d-kserve-676c87888c-ztf48"]
Oct 29 23:01:03.423111 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.423088    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 23:01:03.495635 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495601    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.495828 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495668    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.495828 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495712    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.495828 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495741    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.495828 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495802    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.496098 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495848    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.496098 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495971    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.496098 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496065    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.496098 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.495966    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.496307 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496111    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket\") pod \"a75d478e-e111-427c-b820-a5a33314e693\" (UID: \"a75d478e-e111-427c-b820-a5a33314e693\") "
Oct 29 23:01:03.496361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496319    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.496361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496341    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.496598 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496571    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.496717 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496620    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:03.496843 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496824    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.496994 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496852    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75d478e-e111-427c-b820-a5a33314e693-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.496994 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496866    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.496994 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496879    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.496994 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.496893    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.498802 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.498762    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.498924 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.498850    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.498924 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.498875    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "kube-api-access-7kz5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.499772 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.499754    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo" pod "a75d478e-e111-427c-b820-a5a33314e693" (UID: "a75d478e-e111-427c-b820-a5a33314e693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:03.562226 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.562188    4451 generic.go:356] "Generic (PLEG): container finished" podID="a75d478e-e111-427c-b820-a5a33314e693" containerID="d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387" exitCode=0
Oct 29 23:01:03.562718 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.562247    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"
Oct 29 23:01:03.562718 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.562278    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" event={"ID":"a75d478e-e111-427c-b820-a5a33314e693","Type":"ContainerDied","Data":"d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387"}
Oct 29 23:01:03.562718 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.562330    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt" event={"ID":"a75d478e-e111-427c-b820-a5a33314e693","Type":"ContainerDied","Data":"d9f1b28c74b94d3a20f08b727fd075a4cc5be2d3c2cfc5da93c148014d9072cd"}
Oct 29 23:01:03.562718 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.562363    4451 scope.go:122] "RemoveContainer" containerID="d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387"
Oct 29 23:01:03.574419 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.574399    4451 scope.go:122] "RemoveContainer" containerID="d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387"
Oct 29 23:01:03.574730 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:01:03.574704    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387\": container with ID starting with d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387 not found: ID does not exist" containerID="d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387"
Oct 29 23:01:03.574820 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.574735    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387"} err="failed to get container status \"d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387\": rpc error: code = NotFound desc = could not find container \"d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387\": container with ID starting with d5be4b78784573a27295930a3fadd6ad10c8b5de4f3458bbc347329fe5fcb387 not found: ID does not exist"
Oct 29 23:01:03.598469 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.598442    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75d478e-e111-427c-b820-a5a33314e693-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.598469 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.598466    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kz5c\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-kube-api-access-7kz5c\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.598607 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.598476    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75d478e-e111-427c-b820-a5a33314e693-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.598607 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.598486    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75d478e-e111-427c-b820-a5a33314e693-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.612045 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.612016    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"]
Oct 29 23:01:03.616711 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:03.616690    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-ssfqt"]
Oct 29 23:01:04.163674 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:04.163637    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75d478e-e111-427c-b820-a5a33314e693" path="/var/lib/kubelet/pods/a75d478e-e111-427c-b820-a5a33314e693/volumes"
Oct 29 23:01:04.164318 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:04.164293    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" path="/var/lib/kubelet/pods/de5c8db1-3048-414d-822b-2aa15c83932b/volumes"
Oct 29 23:01:05.981884 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:05.981843    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"]
Oct 29 23:01:05.982401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:05.982105    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy" containerID="cri-o://4bdea5ed81e4834dacbf2d3754fa611a2522b81e68fa723ce21eb93171c84d29" gracePeriod=30
Oct 29 23:01:11.655719 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.655682    4451 generic.go:356] "Generic (PLEG): container finished" podID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerID="4bdea5ed81e4834dacbf2d3754fa611a2522b81e68fa723ce21eb93171c84d29" exitCode=0
Oct 29 23:01:11.656228 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.655752    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" event={"ID":"f0959bf2-4b14-4956-a695-fddffd81a9aa","Type":"ContainerDied","Data":"4bdea5ed81e4834dacbf2d3754fa611a2522b81e68fa723ce21eb93171c84d29"}
Oct 29 23:01:11.689906 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.689884    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 23:01:11.757503 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.757381    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.757503 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.757460    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.757780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.757544    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.757780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.757617    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.757890 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.757783    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.758956    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.758947    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.758988    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.759074    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.759105    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.759101    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.759167    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.759375    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy\") pod \"f0959bf2-4b14-4956-a695-fddffd81a9aa\" (UID: \"f0959bf2-4b14-4956-a695-fddffd81a9aa\") "
Oct 29 23:01:11.759895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.759378    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.760492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.760090    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.760492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.760114    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.760492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.760129    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.760492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.760142    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.760492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.760156    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0959bf2-4b14-4956-a695-fddffd81a9aa-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.761675 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.761412    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.762582 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.762554    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.762700 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.762605    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "kube-api-access-9jvdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.767123 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.763443    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy" pod "f0959bf2-4b14-4956-a695-fddffd81a9aa" (UID: "f0959bf2-4b14-4956-a695-fddffd81a9aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.861560 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.861528    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.861560 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.861557    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.861746 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.861567    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvdk\" (UniqueName: \"kubernetes.io/projected/f0959bf2-4b14-4956-a695-fddffd81a9aa-kube-api-access-9jvdk\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.861746 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:11.861577    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0959bf2-4b14-4956-a695-fddffd81a9aa-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:12.664073 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:12.664037    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj" event={"ID":"f0959bf2-4b14-4956-a695-fddffd81a9aa","Type":"ContainerDied","Data":"f6762a7b77b1ce56f5467b9c3e0fc70edbe62568745ef3cf2749fba0bbf6c0f6"}
Oct 29 23:01:12.664629 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:12.664102    4451 scope.go:122] "RemoveContainer" containerID="4bdea5ed81e4834dacbf2d3754fa611a2522b81e68fa723ce21eb93171c84d29"
Oct 29 23:01:12.664629 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:12.664046    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"
Oct 29 23:01:12.719438 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:12.719406    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"]
Oct 29 23:01:12.729541 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:12.729515    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-55kkj"]
Oct 29 23:01:14.164147 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:14.164106    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" path="/var/lib/kubelet/pods/f0959bf2-4b14-4956-a695-fddffd81a9aa/volumes"
Oct 29 23:01:18.741807 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:01:18.741770    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:01:21.357555 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:21.357523    4451 scope.go:122] "RemoveContainer" containerID="6e85c979e4925c173fca38e2ac383febf353c00fb6feeb18b48fca0b9134a898"
Oct 29 23:01:23.522715 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:23.522677    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"]
Oct 29 23:01:23.523264 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:23.522971    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw" podUID="fcad3545-c689-4336-8fba-719bdd911ae3" containerName="maas-api" containerID="cri-o://0b25017c09ebdb5992ccd874d289cd0fbcecbc0d23d85ed82b1671932ba32078" gracePeriod=30
Oct 29 23:01:23.744415 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:23.744301    4451 generic.go:356] "Generic (PLEG): container finished" podID="fcad3545-c689-4336-8fba-719bdd911ae3" containerID="0b25017c09ebdb5992ccd874d289cd0fbcecbc0d23d85ed82b1671932ba32078" exitCode=0
Oct 29 23:01:23.744415 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:23.744362    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw" event={"ID":"fcad3545-c689-4336-8fba-719bdd911ae3","Type":"ContainerDied","Data":"0b25017c09ebdb5992ccd874d289cd0fbcecbc0d23d85ed82b1671932ba32078"}
Oct 29 23:01:23.955157 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:23.955129    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 23:01:24.032762 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.032725    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs\") pod \"fcad3545-c689-4336-8fba-719bdd911ae3\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") "
Oct 29 23:01:24.033004 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.032840    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg\" (UniqueName: \"kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg\") pod \"fcad3545-c689-4336-8fba-719bdd911ae3\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") "
Oct 29 23:01:24.033004 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.032961    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls\") pod \"fcad3545-c689-4336-8fba-719bdd911ae3\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") "
Oct 29 23:01:24.033139 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.033001    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca\") pod \"fcad3545-c689-4336-8fba-719bdd911ae3\" (UID: \"fcad3545-c689-4336-8fba-719bdd911ae3\") "
Oct 29 23:01:24.034029 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.033961    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca" pod "fcad3545-c689-4336-8fba-719bdd911ae3" (UID: "fcad3545-c689-4336-8fba-719bdd911ae3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:24.036361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.036336    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg" pod "fcad3545-c689-4336-8fba-719bdd911ae3" (UID: "fcad3545-c689-4336-8fba-719bdd911ae3"). InnerVolumeSpecName "kube-api-access-9tqmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:24.049439 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.049392    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs" pod "fcad3545-c689-4336-8fba-719bdd911ae3" (UID: "fcad3545-c689-4336-8fba-719bdd911ae3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:24.051249 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.051223    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls" pod "fcad3545-c689-4336-8fba-719bdd911ae3" (UID: "fcad3545-c689-4336-8fba-719bdd911ae3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:24.071692 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.071657    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"]
Oct 29 23:01:24.072231 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.071944    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" podUID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" containerName="maas-api" containerID="cri-o://ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c" gracePeriod=30
Oct 29 23:01:24.135521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.135481    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.135521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.135521    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqmg\" (UniqueName: \"kubernetes.io/projected/fcad3545-c689-4336-8fba-719bdd911ae3-kube-api-access-9tqmg\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.135789 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.135536    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcad3545-c689-4336-8fba-719bdd911ae3-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.135789 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.135549    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad3545-c689-4336-8fba-719bdd911ae3-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.307037 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.297762    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" podUID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.136:8443/health\": dial tcp 10.133.0.136:8443: connect: connection refused"
Oct 29 23:01:24.550000 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.549964    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 23:01:24.658938 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.647469    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs\") pod \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") "
Oct 29 23:01:24.658938 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.647543    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca\") pod \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") "
Oct 29 23:01:24.658938 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.647590    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\") pod \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") "
Oct 29 23:01:24.658938 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.647763    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l\" (UniqueName: \"kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l\") pod \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\" (UID: \"1851e1e9-8320-40d3-92fe-52d8e8d62ef2\") "
Oct 29 23:01:24.658938 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.653103    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca" pod "1851e1e9-8320-40d3-92fe-52d8e8d62ef2" (UID: "1851e1e9-8320-40d3-92fe-52d8e8d62ef2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:24.749698 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.748301    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l" pod "1851e1e9-8320-40d3-92fe-52d8e8d62ef2" (UID: "1851e1e9-8320-40d3-92fe-52d8e8d62ef2"). InnerVolumeSpecName "kube-api-access-z4b6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:24.752326 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.752292    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.753024 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.752330    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4b6l\" (UniqueName: \"kubernetes.io/projected/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-kube-api-access-z4b6l\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.755971    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls" pod "1851e1e9-8320-40d3-92fe-52d8e8d62ef2" (UID: "1851e1e9-8320-40d3-92fe-52d8e8d62ef2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.756130    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs" pod "1851e1e9-8320-40d3-92fe-52d8e8d62ef2" (UID: "1851e1e9-8320-40d3-92fe-52d8e8d62ef2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.762175    4451 generic.go:356] "Generic (PLEG): container finished" podID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" containerID="ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c" exitCode=0
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.762342    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" event={"ID":"1851e1e9-8320-40d3-92fe-52d8e8d62ef2","Type":"ContainerDied","Data":"ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c"}
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.762393    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c" event={"ID":"1851e1e9-8320-40d3-92fe-52d8e8d62ef2","Type":"ContainerDied","Data":"8be4c3353119a5d0628e9f8bce8101fbd564db83af1c24716a90db6371fec810"}
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.762423    4451 scope.go:122] "RemoveContainer" containerID="ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c"
Oct 29 23:01:24.769092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.765047    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"
Oct 29 23:01:24.781141 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.781078    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw" event={"ID":"fcad3545-c689-4336-8fba-719bdd911ae3","Type":"ContainerDied","Data":"53a75e819bb4ff390dd750d7c7bbcc9079ae733d8847d85c52c4b260e1df66b7"}
Oct 29 23:01:24.781450 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.781391    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"
Oct 29 23:01:24.797514 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.797406    4451 scope.go:122] "RemoveContainer" containerID="ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c"
Oct 29 23:01:24.801525 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:01:24.801306    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c\": container with ID starting with ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c not found: ID does not exist" containerID="ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c"
Oct 29 23:01:24.801525 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.801350    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c"} err="failed to get container status \"ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c\": rpc error: code = NotFound desc = could not find container \"ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c\": container with ID starting with ddd1b939bbedec8d981f7c8e32598fc98557f8720498a739abd3d75cc62f1d0c not found: ID does not exist"
Oct 29 23:01:24.801525 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.801377    4451 scope.go:122] "RemoveContainer" containerID="0b25017c09ebdb5992ccd874d289cd0fbcecbc0d23d85ed82b1671932ba32078"
Oct 29 23:01:24.847800 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.847732    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"]
Oct 29 23:01:24.852188 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.852158    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7be71d-c5c6f85c7-zz42c"]
Oct 29 23:01:24.854059 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.854032    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.854059 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.854056    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1851e1e9-8320-40d3-92fe-52d8e8d62ef2-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.887870 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.887823    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"]
Oct 29 23:01:24.892606 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:24.892577    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b8f54c-85d978579f-w2blw"]
Oct 29 23:01:26.164657 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:26.164624    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" path="/var/lib/kubelet/pods/1851e1e9-8320-40d3-92fe-52d8e8d62ef2/volumes"
Oct 29 23:01:26.165154 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:26.165137    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcad3545-c689-4336-8fba-719bdd911ae3" path="/var/lib/kubelet/pods/fcad3545-c689-4336-8fba-719bdd911ae3/volumes"
Oct 29 23:01:30.705964 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:30.705929    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"]
Oct 29 23:01:45.729364 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:45.729273    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5" podUID="fbf17621-84f3-4c2c-a7a6-b41679a95a08" containerName="main" containerID="cri-o://8295eb49b03819d39f187384e434fff3ac33042e25de2c0ab501a767885a476e" gracePeriod=45
Oct 29 23:01:45.937204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:45.937163    4451 generic.go:356] "Generic (PLEG): container finished" podID="fbf17621-84f3-4c2c-a7a6-b41679a95a08" containerID="8295eb49b03819d39f187384e434fff3ac33042e25de2c0ab501a767885a476e" exitCode=0
Oct 29 23:01:45.937411 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:45.937387    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5" event={"ID":"fbf17621-84f3-4c2c-a7a6-b41679a95a08","Type":"ContainerDied","Data":"8295eb49b03819d39f187384e434fff3ac33042e25de2c0ab501a767885a476e"}
Oct 29 23:01:46.005388 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.005362    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 23:01:46.061028 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.060989    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm\") pod \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") "
Oct 29 23:01:46.061220 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.061110    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache\") pod \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") "
Oct 29 23:01:46.061220 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.061152    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home\") pod \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") "
Oct 29 23:01:46.061348 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.061308    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs\") pod \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") "
Oct 29 23:01:46.061407 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.061390    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir\") pod \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") "
Oct 29 23:01:46.061517 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.061498    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb\" (UniqueName: \"kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb\") pod \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\" (UID: \"fbf17621-84f3-4c2c-a7a6-b41679a95a08\") "
Oct 29 23:01:46.062554 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.062518    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home" pod "fbf17621-84f3-4c2c-a7a6-b41679a95a08" (UID: "fbf17621-84f3-4c2c-a7a6-b41679a95a08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:46.062770 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.062738    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache" pod "fbf17621-84f3-4c2c-a7a6-b41679a95a08" (UID: "fbf17621-84f3-4c2c-a7a6-b41679a95a08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:46.062993 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.062972    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir" pod "fbf17621-84f3-4c2c-a7a6-b41679a95a08" (UID: "fbf17621-84f3-4c2c-a7a6-b41679a95a08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:46.065858 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.065820    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb" pod "fbf17621-84f3-4c2c-a7a6-b41679a95a08" (UID: "fbf17621-84f3-4c2c-a7a6-b41679a95a08"). InnerVolumeSpecName "kube-api-access-krmwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:46.066415 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.066393    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs" pod "fbf17621-84f3-4c2c-a7a6-b41679a95a08" (UID: "fbf17621-84f3-4c2c-a7a6-b41679a95a08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:46.067238 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.067221    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm" pod "fbf17621-84f3-4c2c-a7a6-b41679a95a08" (UID: "fbf17621-84f3-4c2c-a7a6-b41679a95a08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:46.163331 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.163298    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:46.163331 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.163327    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:46.163331 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.163337    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmwb\" (UniqueName: \"kubernetes.io/projected/fbf17621-84f3-4c2c-a7a6-b41679a95a08-kube-api-access-krmwb\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:46.163581 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.163346    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:46.163581 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.163354    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:46.163581 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.163362    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf17621-84f3-4c2c-a7a6-b41679a95a08-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:01:46.945664 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.945630    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"
Oct 29 23:01:46.946164 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.945633    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5" event={"ID":"fbf17621-84f3-4c2c-a7a6-b41679a95a08","Type":"ContainerDied","Data":"699efcf615aadb5881452b83216c2930149e0bd0b3161796c6b3635ed50eb138"}
Oct 29 23:01:46.946164 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.945766    4451 scope.go:122] "RemoveContainer" containerID="8295eb49b03819d39f187384e434fff3ac33042e25de2c0ab501a767885a476e"
Oct 29 23:01:46.991157 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:46.991120    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"]
Oct 29 23:01:47.001805 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:47.001769    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7be71d/ipp-route-7be71d-kserve-bfd576cf7-n8tl5"]
Oct 29 23:01:48.182492 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:01:48.181886    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf17621-84f3-4c2c-a7a6-b41679a95a08" path="/var/lib/kubelet/pods/fbf17621-84f3-4c2c-a7a6-b41679a95a08/volumes"
Oct 29 23:01:48.747374 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:01:48.747337    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:02:05.124814 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:05.124774    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"]
Oct 29 23:02:05.125449 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:05.125116    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy" containerID="cri-o://6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5" gracePeriod=30
Oct 29 23:02:10.398631 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.398572    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 23:02:10.481933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.481889    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482111 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.481977    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482111 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482014    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482111 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482084    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482286 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482178    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482286 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482219    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482286 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482251    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:10.482453 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482354    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.482453 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482280    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482573 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482422    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482573 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482505    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v\") pod \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\" (UID: \"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc\") "
Oct 29 23:02:10.482692 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482585    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.482854 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482833    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.482954 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.482889    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.483445 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.483407    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.483445 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.483438    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.483627 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.483454    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.483627 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.483468    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.483627 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.483483    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.484582 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.484554    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:10.484754 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.484718    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.484943 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.484893    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "kube-api-access-cpw5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:10.485184 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.485168    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token" pod "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" (UID: "d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:10.584732 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.584700    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.584732 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.584729    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.584732 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.584738    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.584998 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:10.584747    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpw5v\" (UniqueName: \"kubernetes.io/projected/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc-kube-api-access-cpw5v\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.154319 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.154284    4451 generic.go:356] "Generic (PLEG): container finished" podID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerID="6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5" exitCode=0
Oct 29 23:02:11.154519 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.154330    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"
Oct 29 23:02:11.154519 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.154360    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" event={"ID":"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc","Type":"ContainerDied","Data":"6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5"}
Oct 29 23:02:11.154519 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.154423    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl" event={"ID":"d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc","Type":"ContainerDied","Data":"a223cfb6e95fb3bf4a2ba542b31499e0db688818283d290a2ceb7fcea5c8439e"}
Oct 29 23:02:11.154519 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.154455    4451 scope.go:122] "RemoveContainer" containerID="6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5"
Oct 29 23:02:11.167683 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.167663    4451 scope.go:122] "RemoveContainer" containerID="6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5"
Oct 29 23:02:11.168156 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:11.168125    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5\": container with ID starting with 6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5 not found: ID does not exist" containerID="6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5"
Oct 29 23:02:11.168268 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.168166    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5"} err="failed to get container status \"6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5\": rpc error: code = NotFound desc = could not find container \"6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5\": container with ID starting with 6ac5150029bf894db3aed0e9091ff104a851a5eb2db6bc9df3818c2841a5c3c5 not found: ID does not exist"
Oct 29 23:02:11.213130 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.213100    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"]
Oct 29 23:02:11.223984 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:11.223958    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jwrdl"]
Oct 29 23:02:12.165184 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:12.165153    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" path="/var/lib/kubelet/pods/d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc/volumes"
Oct 29 23:02:13.283579 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:13.283542    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"]
Oct 29 23:02:13.284140 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:13.283778    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy" containerID="cri-o://7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62" gracePeriod=30
Oct 29 23:02:18.644581 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.644548    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 23:02:18.733521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.733481    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.733830 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.733808    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734014 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.733987    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734143 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.734076    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734209 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.734171    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734303 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.734285    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734362 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.734351    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734424 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.734380    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.734482 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.734426    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs\") pod \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\" (UID: \"ff36cacd-fec0-41a2-aa34-d05f20689ffc\") "
Oct 29 23:02:18.735253 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.735229    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.736004 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.735983    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.736440 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.736417    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:18.738031 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.737995    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.738351 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.738330    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.743865 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.743830    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:18.747076 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.747051    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:18.747352 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.747326    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.747559 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.747537    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6" pod "ff36cacd-fec0-41a2-aa34-d05f20689ffc" (UID: "ff36cacd-fec0-41a2-aa34-d05f20689ffc"). InnerVolumeSpecName "kube-api-access-mmlb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:18.761401 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:18.761373    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.836869    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.836959    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.836973    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.836985    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.836996    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.837007    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.837019    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmlb6\" (UniqueName: \"kubernetes.io/projected/ff36cacd-fec0-41a2-aa34-d05f20689ffc-kube-api-access-mmlb6\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.837033    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.837623 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:18.837045    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff36cacd-fec0-41a2-aa34-d05f20689ffc-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.210018 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.209983    4451 generic.go:356] "Generic (PLEG): container finished" podID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerID="7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62" exitCode=0
Oct 29 23:02:19.210219 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.210034    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"
Oct 29 23:02:19.210219 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.210067    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" event={"ID":"ff36cacd-fec0-41a2-aa34-d05f20689ffc","Type":"ContainerDied","Data":"7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62"}
Oct 29 23:02:19.210219 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.210126    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp" event={"ID":"ff36cacd-fec0-41a2-aa34-d05f20689ffc","Type":"ContainerDied","Data":"7111ed049a66581c8b867ca9e0f9ddc510f3a153d114706823c76f9817711e4c"}
Oct 29 23:02:19.210219 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.210156    4451 scope.go:122] "RemoveContainer" containerID="7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62"
Oct 29 23:02:19.220701 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.220679    4451 scope.go:122] "RemoveContainer" containerID="7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62"
Oct 29 23:02:19.220968 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:19.220949    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62\": container with ID starting with 7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62 not found: ID does not exist" containerID="7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62"
Oct 29 23:02:19.221047 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.220981    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62"} err="failed to get container status \"7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62\": rpc error: code = NotFound desc = could not find container \"7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62\": container with ID starting with 7adcec2a1b9ed6bea89fd2f0c459a7b28200a9d50a7c77e94b05c8cd2c9ecc62 not found: ID does not exist"
Oct 29 23:02:19.259298 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.259216    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"]
Oct 29 23:02:19.268856 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:19.268833    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-w9vdp"]
Oct 29 23:02:20.164300 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:20.164266    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" path="/var/lib/kubelet/pods/ff36cacd-fec0-41a2-aa34-d05f20689ffc/volumes"
Oct 29 23:02:34.932036 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:34.931946    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"]
Oct 29 23:02:34.932539 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:34.932185    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b" podUID="770b4385-960b-4f68-9532-483d0dae622c" containerName="maas-api" containerID="cri-o://22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2" gracePeriod=30
Oct 29 23:02:35.281686 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.281657    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 23:02:35.287220 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.287183    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca\") pod \"770b4385-960b-4f68-9532-483d0dae622c\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") "
Oct 29 23:02:35.287360 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.287301    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp\" (UniqueName: \"kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp\") pod \"770b4385-960b-4f68-9532-483d0dae622c\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") "
Oct 29 23:02:35.287425 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.287397    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs\") pod \"770b4385-960b-4f68-9532-483d0dae622c\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") "
Oct 29 23:02:35.287482 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.287467    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls\") pod \"770b4385-960b-4f68-9532-483d0dae622c\" (UID: \"770b4385-960b-4f68-9532-483d0dae622c\") "
Oct 29 23:02:35.287542 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.287487    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca" pod "770b4385-960b-4f68-9532-483d0dae622c" (UID: "770b4385-960b-4f68-9532-483d0dae622c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:35.288019 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.287994    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770b4385-960b-4f68-9532-483d0dae622c-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.290270 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.290147    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs" pod "770b4385-960b-4f68-9532-483d0dae622c" (UID: "770b4385-960b-4f68-9532-483d0dae622c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:35.290270 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.290179    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp" pod "770b4385-960b-4f68-9532-483d0dae622c" (UID: "770b4385-960b-4f68-9532-483d0dae622c"). InnerVolumeSpecName "kube-api-access-nl4fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:35.290270 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.290237    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls" pod "770b4385-960b-4f68-9532-483d0dae622c" (UID: "770b4385-960b-4f68-9532-483d0dae622c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:35.398277 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.397932    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.398277 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.397977    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770b4385-960b-4f68-9532-483d0dae622c-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.398277 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.397992    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl4fp\" (UniqueName: \"kubernetes.io/projected/770b4385-960b-4f68-9532-483d0dae622c-kube-api-access-nl4fp\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.443990 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.441438    4451 generic.go:356] "Generic (PLEG): container finished" podID="770b4385-960b-4f68-9532-483d0dae622c" containerID="22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2" exitCode=0
Oct 29 23:02:35.443990 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.441612    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b" event={"ID":"770b4385-960b-4f68-9532-483d0dae622c","Type":"ContainerDied","Data":"22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2"}
Oct 29 23:02:35.443990 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.441691    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b" event={"ID":"770b4385-960b-4f68-9532-483d0dae622c","Type":"ContainerDied","Data":"cb758a8436753b0e68cae73b813b6dc5bc33a957067d5a53cdcc400920ecf930"}
Oct 29 23:02:35.443990 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.441727    4451 scope.go:122] "RemoveContainer" containerID="22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2"
Oct 29 23:02:35.443990 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.441884    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"
Oct 29 23:02:35.475616 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.475590    4451 scope.go:122] "RemoveContainer" containerID="22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2"
Oct 29 23:02:35.475902 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:35.475884    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2\": container with ID starting with 22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2 not found: ID does not exist" containerID="22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2"
Oct 29 23:02:35.475995 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.475945    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2"} err="failed to get container status \"22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2\": rpc error: code = NotFound desc = could not find container \"22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2\": container with ID starting with 22e1505b163c8d20ab00421306d76d51ec981b043f52c51604b34689f09372b2 not found: ID does not exist"
Oct 29 23:02:35.498226 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.498188    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"]
Oct 29 23:02:35.501978 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:35.501953    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eb02df-6b95fbd44-64m5b"]
Oct 29 23:02:36.163812 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:36.163774    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770b4385-960b-4f68-9532-483d0dae622c" path="/var/lib/kubelet/pods/770b4385-960b-4f68-9532-483d0dae622c/volumes"
Oct 29 23:02:38.080599 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:38.080548    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"]
Oct 29 23:02:38.081982 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:38.081704    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy" containerID="cri-o://01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3" gracePeriod=30
Oct 29 23:02:42.479094 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:42.466479    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"]
Oct 29 23:02:42.479094 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:42.466713    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy" containerID="cri-o://f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec" gracePeriod=30
Oct 29 23:02:42.693515 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:42.692820    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"]
Oct 29 23:02:42.693515 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:42.693106    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" podUID="41d9ebb5-d732-4eb5-86a7-f088a59bfd04" containerName="maas-api" containerID="cri-o://d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e" gracePeriod=30
Oct 29 23:02:43.052346 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.052318    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 23:02:43.131085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.131051    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca\") pod \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") "
Oct 29 23:02:43.131234 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.131214    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\") pod \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") "
Oct 29 23:02:43.131312 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.131264    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52\" (UniqueName: \"kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52\") pod \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") "
Oct 29 23:02:43.131491 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.131466    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs\") pod \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\" (UID: \"41d9ebb5-d732-4eb5-86a7-f088a59bfd04\") "
Oct 29 23:02:43.132077 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.131738    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca" pod "41d9ebb5-d732-4eb5-86a7-f088a59bfd04" (UID: "41d9ebb5-d732-4eb5-86a7-f088a59bfd04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.132204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.132129    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.134311 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.134283    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls" pod "41d9ebb5-d732-4eb5-86a7-f088a59bfd04" (UID: "41d9ebb5-d732-4eb5-86a7-f088a59bfd04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:43.134500 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.134469    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52" pod "41d9ebb5-d732-4eb5-86a7-f088a59bfd04" (UID: "41d9ebb5-d732-4eb5-86a7-f088a59bfd04"). InnerVolumeSpecName "kube-api-access-cdn52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.134648 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.134627    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs" pod "41d9ebb5-d732-4eb5-86a7-f088a59bfd04" (UID: "41d9ebb5-d732-4eb5-86a7-f088a59bfd04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:43.242933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.240961    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.242933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.241005    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.242933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.241019    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdn52\" (UniqueName: \"kubernetes.io/projected/41d9ebb5-d732-4eb5-86a7-f088a59bfd04-kube-api-access-cdn52\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.421707 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.421671    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 23:02:43.517974 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.517874    4451 generic.go:356] "Generic (PLEG): container finished" podID="41d9ebb5-d732-4eb5-86a7-f088a59bfd04" containerID="d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e" exitCode=0
Oct 29 23:02:43.518730 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.518096    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" event={"ID":"41d9ebb5-d732-4eb5-86a7-f088a59bfd04","Type":"ContainerDied","Data":"d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e"}
Oct 29 23:02:43.518730 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.518155    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp" event={"ID":"41d9ebb5-d732-4eb5-86a7-f088a59bfd04","Type":"ContainerDied","Data":"5ced344b595f8f03fc0fa8beca78dcbdae9d0b2aba69032d10e991f8dfc6ccdc"}
Oct 29 23:02:43.518730 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.518187    4451 scope.go:122] "RemoveContainer" containerID="d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e"
Oct 29 23:02:43.518730 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.518347    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"
Oct 29 23:02:43.547006 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.546891    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547195 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547104    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547195 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547140    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547335 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547259    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547335 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547290    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547425 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547372    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547481 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547427    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547481 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547466    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.547589 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.547501    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket\") pod \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\" (UID: \"cb53b1d8-2f2b-41c9-88f4-58ca148122be\") "
Oct 29 23:02:43.548338 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.548288    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.551057 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.551024    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "kube-api-access-ps6cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.551260 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.551239    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.551787 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.551760    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.552057 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.552018    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.552204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.552178    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.553751 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.553721    4451 generic.go:356] "Generic (PLEG): container finished" podID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerID="01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3" exitCode=0
Oct 29 23:02:43.553858 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.553793    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" event={"ID":"cb53b1d8-2f2b-41c9-88f4-58ca148122be","Type":"ContainerDied","Data":"01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3"}
Oct 29 23:02:43.553858 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.553845    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs" event={"ID":"cb53b1d8-2f2b-41c9-88f4-58ca148122be","Type":"ContainerDied","Data":"82b7f25a82834d7f4dfd55c7b795ce0755bd6dc70dc56854eef39aa92a204af1"}
Oct 29 23:02:43.554031 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.554017    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"
Oct 29 23:02:43.555422 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.555384    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.556263 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.556129    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.558150 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.558016    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo" pod "cb53b1d8-2f2b-41c9-88f4-58ca148122be" (UID: "cb53b1d8-2f2b-41c9-88f4-58ca148122be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:43.567078 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.567051    4451 scope.go:122] "RemoveContainer" containerID="d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e"
Oct 29 23:02:43.567407 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:43.567382    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e\": container with ID starting with d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e not found: ID does not exist" containerID="d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e"
Oct 29 23:02:43.567497 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.567419    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e"} err="failed to get container status \"d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e\": rpc error: code = NotFound desc = could not find container \"d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e\": container with ID starting with d67ddfe6759ef644de7fd7319c5f717fd8e12cd88cb92cdbbf60d57c8cba891e not found: ID does not exist"
Oct 29 23:02:43.567497 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.567444    4451 scope.go:122] "RemoveContainer" containerID="01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3"
Oct 29 23:02:43.577257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.577230    4451 scope.go:122] "RemoveContainer" containerID="01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3"
Oct 29 23:02:43.577503 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:43.577483    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3\": container with ID starting with 01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3 not found: ID does not exist" containerID="01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3"
Oct 29 23:02:43.577610 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.577508    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3"} err="failed to get container status \"01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3\": rpc error: code = NotFound desc = could not find container \"01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3\": container with ID starting with 01b27fa5459313d778942c0d0fdc28155715258980808f5a8ee363c6ce7210e3 not found: ID does not exist"
Oct 29 23:02:43.593278 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.593214    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"]
Oct 29 23:02:43.596000 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.595981    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85862e-6d888f96b-2ftfp"]
Oct 29 23:02:43.649570 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649535    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649570 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649570    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649586    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649601    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649613    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649627    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649641    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb53b1d8-2f2b-41c9-88f4-58ca148122be-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649653    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb53b1d8-2f2b-41c9-88f4-58ca148122be-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.649782 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.649667    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps6cw\" (UniqueName: \"kubernetes.io/projected/cb53b1d8-2f2b-41c9-88f4-58ca148122be-kube-api-access-ps6cw\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.898765 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.898734    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"]
Oct 29 23:02:43.903698 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:43.903672    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jmfgs"]
Oct 29 23:02:44.164715 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:44.164630    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d9ebb5-d732-4eb5-86a7-f088a59bfd04" path="/var/lib/kubelet/pods/41d9ebb5-d732-4eb5-86a7-f088a59bfd04/volumes"
Oct 29 23:02:44.165223 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:44.165205    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" path="/var/lib/kubelet/pods/cb53b1d8-2f2b-41c9-88f4-58ca148122be/volumes"
Oct 29 23:02:47.741314 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.741256    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 23:02:47.905119 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905091    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905323 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905158    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905323 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905303    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905453 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905418    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905510 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905461    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905566 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905515    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905622 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905560    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905622 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905614    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.905757 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.905676    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy\") pod \"9e810219-babe-4d39-9b85-66d07fd14df6\" (UID: \"9e810219-babe-4d39-9b85-66d07fd14df6\") "
Oct 29 23:02:47.906334 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.906010    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.906334 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.906269    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:47.906334 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.906279    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:47.906334 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.906324    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.906687 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.906390    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.906687 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.906487    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.907546 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.907527    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "kube-api-access-v2btl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:47.907812 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.907793    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:47.908150 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.908132    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:47.908398 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:47.908383    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy" pod "9e810219-babe-4d39-9b85-66d07fd14df6" (UID: "9e810219-babe-4d39-9b85-66d07fd14df6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007510    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007540    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007551    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e810219-babe-4d39-9b85-66d07fd14df6-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007559    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007568    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007575    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e810219-babe-4d39-9b85-66d07fd14df6-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007584    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2btl\" (UniqueName: \"kubernetes.io/projected/9e810219-babe-4d39-9b85-66d07fd14df6-kube-api-access-v2btl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.007907 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.007594    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e810219-babe-4d39-9b85-66d07fd14df6-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.587357 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.587315    4451 generic.go:356] "Generic (PLEG): container finished" podID="9e810219-babe-4d39-9b85-66d07fd14df6" containerID="f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec" exitCode=0
Oct 29 23:02:48.587577 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.587388    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"
Oct 29 23:02:48.587577 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.587434    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" event={"ID":"9e810219-babe-4d39-9b85-66d07fd14df6","Type":"ContainerDied","Data":"f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec"}
Oct 29 23:02:48.587577 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.587496    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd" event={"ID":"9e810219-babe-4d39-9b85-66d07fd14df6","Type":"ContainerDied","Data":"b1490afa6ad682d68435c68e348c6c22515777277d19e9e0daffddc8881dec41"}
Oct 29 23:02:48.587577 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.587521    4451 scope.go:122] "RemoveContainer" containerID="f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec"
Oct 29 23:02:48.597644 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.597443    4451 scope.go:122] "RemoveContainer" containerID="f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec"
Oct 29 23:02:48.597761 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:48.597741    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec\": container with ID starting with f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec not found: ID does not exist" containerID="f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec"
Oct 29 23:02:48.597813 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.597769    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec"} err="failed to get container status \"f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec\": rpc error: code = NotFound desc = could not find container \"f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec\": container with ID starting with f70f4eaacb32f1be463a505ee21d5200f2fc9a3b4096e91a37145dae6ba775ec not found: ID does not exist"
Oct 29 23:02:48.645515 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.645477    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"]
Oct 29 23:02:48.655783 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:48.655750    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-8j4dd"]
Oct 29 23:02:48.742648 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:48.742611    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:02:50.163633 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:50.163596    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" path="/var/lib/kubelet/pods/9e810219-babe-4d39-9b85-66d07fd14df6/volumes"
Oct 29 23:02:57.150671 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.150623    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"]
Oct 29 23:02:57.151316 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.150826    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5" podUID="be906733-a9c7-4738-99ee-40d41b4824a2" containerName="maas-api" containerID="cri-o://d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444" gracePeriod=30
Oct 29 23:02:57.472629 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.465164    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 23:02:57.579354 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.579318    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs\") pod \"be906733-a9c7-4738-99ee-40d41b4824a2\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") "
Oct 29 23:02:57.579533 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.579452    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls\") pod \"be906733-a9c7-4738-99ee-40d41b4824a2\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") "
Oct 29 23:02:57.579582 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.579558    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4\" (UniqueName: \"kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4\") pod \"be906733-a9c7-4738-99ee-40d41b4824a2\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") "
Oct 29 23:02:57.579676 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.579659    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca\") pod \"be906733-a9c7-4738-99ee-40d41b4824a2\" (UID: \"be906733-a9c7-4738-99ee-40d41b4824a2\") "
Oct 29 23:02:57.580120 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.580084    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca" pod "be906733-a9c7-4738-99ee-40d41b4824a2" (UID: "be906733-a9c7-4738-99ee-40d41b4824a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:57.580357 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.580334    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be906733-a9c7-4738-99ee-40d41b4824a2-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:57.581722 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.581693    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls" pod "be906733-a9c7-4738-99ee-40d41b4824a2" (UID: "be906733-a9c7-4738-99ee-40d41b4824a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:57.581831 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.581739    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs" pod "be906733-a9c7-4738-99ee-40d41b4824a2" (UID: "be906733-a9c7-4738-99ee-40d41b4824a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:57.581831 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.581776    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4" pod "be906733-a9c7-4738-99ee-40d41b4824a2" (UID: "be906733-a9c7-4738-99ee-40d41b4824a2"). InnerVolumeSpecName "kube-api-access-5zzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:57.687039 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.682693    4451 generic.go:356] "Generic (PLEG): container finished" podID="be906733-a9c7-4738-99ee-40d41b4824a2" containerID="d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444" exitCode=0
Oct 29 23:02:57.687039 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.682970    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5" event={"ID":"be906733-a9c7-4738-99ee-40d41b4824a2","Type":"ContainerDied","Data":"d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444"}
Oct 29 23:02:57.687039 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.683047    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5" event={"ID":"be906733-a9c7-4738-99ee-40d41b4824a2","Type":"ContainerDied","Data":"b93b00bd070f9c96cd75991e8504757ee1fa54ced6a0c5a9f3d9aae811111037"}
Oct 29 23:02:57.687039 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.683091    4451 scope.go:122] "RemoveContainer" containerID="d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444"
Oct 29 23:02:57.687039 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.683304    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"
Oct 29 23:02:57.691451 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.691407    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:57.691451 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.691440    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzh4\" (UniqueName: \"kubernetes.io/projected/be906733-a9c7-4738-99ee-40d41b4824a2-kube-api-access-5zzh4\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:57.691451 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.691453    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be906733-a9c7-4738-99ee-40d41b4824a2-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:02:57.704617 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.704591    4451 scope.go:122] "RemoveContainer" containerID="d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444"
Oct 29 23:02:57.704998 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:02:57.704895    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444\": container with ID starting with d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444 not found: ID does not exist" containerID="d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444"
Oct 29 23:02:57.704998 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.704956    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444"} err="failed to get container status \"d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444\": rpc error: code = NotFound desc = could not find container \"d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444\": container with ID starting with d62734d17d9e8e3fff826662edccc5db1ce52df5877cb5fb0efcf501b1dc6444 not found: ID does not exist"
Oct 29 23:02:57.741473 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.741443    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"]
Oct 29 23:02:57.755016 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:57.754977    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02720a-577b5bbd8-kbdl5"]
Oct 29 23:02:58.165486 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:58.165252    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be906733-a9c7-4738-99ee-40d41b4824a2" path="/var/lib/kubelet/pods/be906733-a9c7-4738-99ee-40d41b4824a2/volumes"
Oct 29 23:02:58.269904 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:58.269856    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"]
Oct 29 23:02:58.270137 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:02:58.270091    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy" containerID="cri-o://638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b" gracePeriod=30
Oct 29 23:03:03.537104 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.537036    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 23:03:03.663152 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663120    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663343 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663216    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663432 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663374    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663432 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663424    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663561 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663453    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663561 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663535    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663711 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663591    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663711 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663644    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:03.663711 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663663    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.663711 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663572    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:03.663890 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663752    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:03.663890 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663752    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:03.663890 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.663808    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket\") pod \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\" (UID: \"c316150f-802e-4b9e-97eb-2611bfdf2ae3\") "
Oct 29 23:03:03.664161 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.664129    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:03.664383 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.664364    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.664435 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.664389    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.664435 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.664404    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.664435 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.664418    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.664435 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.664431    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.665777 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.665745    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:03.665956 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.665933    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:03.665956 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.665949    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "kube-api-access-ltwld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:03.666081 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.666065    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy" pod "c316150f-802e-4b9e-97eb-2611bfdf2ae3" (UID: "c316150f-802e-4b9e-97eb-2611bfdf2ae3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:03.732358 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.732319    4451 generic.go:356] "Generic (PLEG): container finished" podID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerID="638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b" exitCode=0
Oct 29 23:03:03.732510 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.732374    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"
Oct 29 23:03:03.732510 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.732400    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" event={"ID":"c316150f-802e-4b9e-97eb-2611bfdf2ae3","Type":"ContainerDied","Data":"638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b"}
Oct 29 23:03:03.732510 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.732493    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z" event={"ID":"c316150f-802e-4b9e-97eb-2611bfdf2ae3","Type":"ContainerDied","Data":"a741b8a34ff1ef43be72c9000abe8d91bd4bc121f09e0e3fbc6c7cff6ecb2ef8"}
Oct 29 23:03:03.732666 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.732522    4451 scope.go:122] "RemoveContainer" containerID="638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b"
Oct 29 23:03:03.744462 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.744438    4451 scope.go:122] "RemoveContainer" containerID="638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b"
Oct 29 23:03:03.744800 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:03.744781    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b\": container with ID starting with 638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b not found: ID does not exist" containerID="638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b"
Oct 29 23:03:03.744870 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.744809    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b"} err="failed to get container status \"638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b\": rpc error: code = NotFound desc = could not find container \"638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b\": container with ID starting with 638e368860035b05302710d9ce8f850e91e8165cad631d8613170aa7fcf8933b not found: ID does not exist"
Oct 29 23:03:03.766008 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.765983    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwld\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-kube-api-access-ltwld\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.766008 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.766006    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.766142 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.766015    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.766142 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.766023    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c316150f-802e-4b9e-97eb-2611bfdf2ae3-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:03.791445 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.791380    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"]
Oct 29 23:03:03.800240 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:03.800187    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-bkj9z"]
Oct 29 23:03:04.163774 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:04.163741    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" path="/var/lib/kubelet/pods/c316150f-802e-4b9e-97eb-2611bfdf2ae3/volumes"
Oct 29 23:03:06.682124 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:06.682087    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"]
Oct 29 23:03:06.682644 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:06.682364    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx" podUID="a038013b-5758-420e-a8f0-05ba413a1fcb" containerName="maas-api" containerID="cri-o://b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa" gracePeriod=30
Oct 29 23:03:07.062199 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.062153    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 23:03:07.150282 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.150244    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9\" (UniqueName: \"kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9\") pod \"a038013b-5758-420e-a8f0-05ba413a1fcb\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") "
Oct 29 23:03:07.150476 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.150311    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls\") pod \"a038013b-5758-420e-a8f0-05ba413a1fcb\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") "
Oct 29 23:03:07.150476 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.150348    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs\") pod \"a038013b-5758-420e-a8f0-05ba413a1fcb\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") "
Oct 29 23:03:07.150579 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.150481    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca\") pod \"a038013b-5758-420e-a8f0-05ba413a1fcb\" (UID: \"a038013b-5758-420e-a8f0-05ba413a1fcb\") "
Oct 29 23:03:07.150898 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.150869    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca" pod "a038013b-5758-420e-a8f0-05ba413a1fcb" (UID: "a038013b-5758-420e-a8f0-05ba413a1fcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:07.151097 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.151077    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a038013b-5758-420e-a8f0-05ba413a1fcb-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:07.153116 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.153079    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9" pod "a038013b-5758-420e-a8f0-05ba413a1fcb" (UID: "a038013b-5758-420e-a8f0-05ba413a1fcb"). InnerVolumeSpecName "kube-api-access-2wml9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:07.153204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.153110    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls" pod "a038013b-5758-420e-a8f0-05ba413a1fcb" (UID: "a038013b-5758-420e-a8f0-05ba413a1fcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:07.153204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.153170    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs" pod "a038013b-5758-420e-a8f0-05ba413a1fcb" (UID: "a038013b-5758-420e-a8f0-05ba413a1fcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:07.252780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.252740    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wml9\" (UniqueName: \"kubernetes.io/projected/a038013b-5758-420e-a8f0-05ba413a1fcb-kube-api-access-2wml9\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:07.252780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.252774    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:07.252780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.252787    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a038013b-5758-420e-a8f0-05ba413a1fcb-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:07.765246 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.765211    4451 generic.go:356] "Generic (PLEG): container finished" podID="a038013b-5758-420e-a8f0-05ba413a1fcb" containerID="b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa" exitCode=0
Oct 29 23:03:07.765685 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.765264    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"
Oct 29 23:03:07.765685 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.765298    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx" event={"ID":"a038013b-5758-420e-a8f0-05ba413a1fcb","Type":"ContainerDied","Data":"b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa"}
Oct 29 23:03:07.765685 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.765356    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx" event={"ID":"a038013b-5758-420e-a8f0-05ba413a1fcb","Type":"ContainerDied","Data":"5ab1f7f7e78944f7ab88337fe98a5dfcafbd8cf5307677e56105656532b382f9"}
Oct 29 23:03:07.765685 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.765381    4451 scope.go:122] "RemoveContainer" containerID="b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa"
Oct 29 23:03:07.776269 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.776249    4451 scope.go:122] "RemoveContainer" containerID="b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa"
Oct 29 23:03:07.776550 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:07.776528    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa\": container with ID starting with b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa not found: ID does not exist" containerID="b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa"
Oct 29 23:03:07.776622 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.776558    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa"} err="failed to get container status \"b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa\": rpc error: code = NotFound desc = could not find container \"b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa\": container with ID starting with b558e5deccd99ac1cad38923538288a9db9ccc4a69ba893ae8cae401e789afaa not found: ID does not exist"
Oct 29 23:03:07.811534 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.811500    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"]
Oct 29 23:03:07.821845 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:07.821818    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3ee017-59c58f6475-j4rqx"]
Oct 29 23:03:08.164724 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:08.164692    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a038013b-5758-420e-a8f0-05ba413a1fcb" path="/var/lib/kubelet/pods/a038013b-5758-420e-a8f0-05ba413a1fcb/volumes"
Oct 29 23:03:10.186899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:10.186863    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"]
Oct 29 23:03:10.197373 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:10.197343    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"]
Oct 29 23:03:10.205571 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:10.205542    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"]
Oct 29 23:03:14.379266 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:14.379228    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"]
Oct 29 23:03:14.379661 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:14.379559    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" containerID="cri-o://02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2" gracePeriod=30
Oct 29 23:03:18.740160 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:18.740123    4451 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:18.740691 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:18.740181    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:18.742372 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:18.742339    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:03:19.665063 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.665037    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 23:03:19.725819 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.725784    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.726008 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.725850    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.726093 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726050    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.726160 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726108    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.726210 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726162    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.726269 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726245    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726365    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726406    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726460    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726552    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket\") pod \"7598412d-8b77-48fb-8d20-ee60f736228e\" (UID: \"7598412d-8b77-48fb-8d20-ee60f736228e\") "
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.726752    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.727208    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.727232    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.727350    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.727517    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:19.727651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.727599    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.728744 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.728705    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.728744 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.728728    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.729307 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.729281    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:19.730894 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.730862    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl" pod "7598412d-8b77-48fb-8d20-ee60f736228e" (UID: "7598412d-8b77-48fb-8d20-ee60f736228e"). InnerVolumeSpecName "kube-api-access-7r6rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.828349 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828312    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7598412d-8b77-48fb-8d20-ee60f736228e-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.828349 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828340    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.828349 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828349    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.828862 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828360    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.828862 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828369    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6rl\" (UniqueName: \"kubernetes.io/projected/7598412d-8b77-48fb-8d20-ee60f736228e-kube-api-access-7r6rl\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.828862 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828377    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7598412d-8b77-48fb-8d20-ee60f736228e-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.828862 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.828385    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7598412d-8b77-48fb-8d20-ee60f736228e-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.837132 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.837104    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"]
Oct 29 23:03:19.844427 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.844399    4451 generic.go:356] "Generic (PLEG): container finished" podID="7598412d-8b77-48fb-8d20-ee60f736228e" containerID="02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2" exitCode=0
Oct 29 23:03:19.844537 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.844466    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" event={"ID":"7598412d-8b77-48fb-8d20-ee60f736228e","Type":"ContainerDied","Data":"02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2"}
Oct 29 23:03:19.844537 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.844513    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8" event={"ID":"7598412d-8b77-48fb-8d20-ee60f736228e","Type":"ContainerDied","Data":"bb8a20fc7c048b906957baf955532e3c5532ce7a4eac19a987d83c26f5ec265b"}
Oct 29 23:03:19.844537 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.844535    4451 scope.go:122] "RemoveContainer" containerID="02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2"
Oct 29 23:03:19.844730 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.844705    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"
Oct 29 23:03:19.862803 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.862762    4451 scope.go:122] "RemoveContainer" containerID="02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2"
Oct 29 23:03:19.863377 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:19.863348    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2\": container with ID starting with 02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2 not found: ID does not exist" containerID="02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2"
Oct 29 23:03:19.863485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.863386    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2"} err="failed to get container status \"02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2\": rpc error: code = NotFound desc = could not find container \"02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2\": container with ID starting with 02dac708ed612bf1ccceedefdafce0e469378fcc7638199b23ef187ccb2f05a2 not found: ID does not exist"
Oct 29 23:03:19.910014 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.909773    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"]
Oct 29 23:03:19.912721 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:19.912691    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hdcz8"]
Oct 29 23:03:20.164367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:20.164287    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" path="/var/lib/kubelet/pods/7598412d-8b77-48fb-8d20-ee60f736228e/volumes"
Oct 29 23:03:25.211386 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.211340    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" containerID="cri-o://a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1" gracePeriod=45
Oct 29 23:03:25.220086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.220041    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" containerID="cri-o://ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6" gracePeriod=45
Oct 29 23:03:25.227306 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.227254    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" containerID="cri-o://fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780" gracePeriod=45
Oct 29 23:03:25.245900 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.245851    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": dial tcp 10.133.0.59:8000: connect: connection refused"
Oct 29 23:03:25.526142 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.526107    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 23:03:25.553526 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.553497    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 23:03:25.608511 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.608485    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 23:03:25.609138 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.608480    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home\") pod \"4c631028-0a42-445c-be49-b8aff3255b33\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") "
Oct 29 23:03:25.609340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.609322    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm\") pod \"4c631028-0a42-445c-be49-b8aff3255b33\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") "
Oct 29 23:03:25.609495 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.609479    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir\") pod \"4c631028-0a42-445c-be49-b8aff3255b33\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") "
Oct 29 23:03:25.609669 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.609650    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache\") pod \"4c631028-0a42-445c-be49-b8aff3255b33\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") "
Oct 29 23:03:25.609854 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.609837    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7\" (UniqueName: \"kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7\") pod \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") "
Oct 29 23:03:25.610026 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610010    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm\") pod \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") "
Oct 29 23:03:25.610142 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610129    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs\") pod \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") "
Oct 29 23:03:25.610238 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610228    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp\" (UniqueName: \"kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp\") pod \"4c631028-0a42-445c-be49-b8aff3255b33\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") "
Oct 29 23:03:25.610356 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610345    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir\") pod \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") "
Oct 29 23:03:25.610428 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610419    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs\") pod \"4c631028-0a42-445c-be49-b8aff3255b33\" (UID: \"4c631028-0a42-445c-be49-b8aff3255b33\") "
Oct 29 23:03:25.610529 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610516    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache\") pod \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") "
Oct 29 23:03:25.610684 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.610670    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home\") pod \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\" (UID: \"ebdd9e25-f8f5-45f4-aafe-13ac66426e22\") "
Oct 29 23:03:25.611558 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.608761    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home" pod "4c631028-0a42-445c-be49-b8aff3255b33" (UID: "4c631028-0a42-445c-be49-b8aff3255b33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.611662 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.611540    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home" pod "ebdd9e25-f8f5-45f4-aafe-13ac66426e22" (UID: "ebdd9e25-f8f5-45f4-aafe-13ac66426e22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.612495 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.612465    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache" pod "4c631028-0a42-445c-be49-b8aff3255b33" (UID: "4c631028-0a42-445c-be49-b8aff3255b33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.612712 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.612690    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir" pod "4c631028-0a42-445c-be49-b8aff3255b33" (UID: "4c631028-0a42-445c-be49-b8aff3255b33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.614559 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.614532    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir" pod "ebdd9e25-f8f5-45f4-aafe-13ac66426e22" (UID: "ebdd9e25-f8f5-45f4-aafe-13ac66426e22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.617320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.617290    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs" pod "4c631028-0a42-445c-be49-b8aff3255b33" (UID: "4c631028-0a42-445c-be49-b8aff3255b33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:25.617552 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.617531    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache" pod "ebdd9e25-f8f5-45f4-aafe-13ac66426e22" (UID: "ebdd9e25-f8f5-45f4-aafe-13ac66426e22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.621308 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.621273    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp" pod "4c631028-0a42-445c-be49-b8aff3255b33" (UID: "4c631028-0a42-445c-be49-b8aff3255b33"). InnerVolumeSpecName "kube-api-access-cmcrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:25.625937 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.625884    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm" pod "4c631028-0a42-445c-be49-b8aff3255b33" (UID: "4c631028-0a42-445c-be49-b8aff3255b33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.627328 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.627276    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm" pod "ebdd9e25-f8f5-45f4-aafe-13ac66426e22" (UID: "ebdd9e25-f8f5-45f4-aafe-13ac66426e22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.627507 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.627491    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs" pod "ebdd9e25-f8f5-45f4-aafe-13ac66426e22" (UID: "ebdd9e25-f8f5-45f4-aafe-13ac66426e22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:25.627734 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.627708    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7" pod "ebdd9e25-f8f5-45f4-aafe-13ac66426e22" (UID: "ebdd9e25-f8f5-45f4-aafe-13ac66426e22"). InnerVolumeSpecName "kube-api-access-nnsw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:25.712666 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712629    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache\") pod \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") "
Oct 29 23:03:25.712666 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712664    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm\") pod \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") "
Oct 29 23:03:25.712999 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712803    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir\") pod \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") "
Oct 29 23:03:25.712999 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712840    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home\") pod \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") "
Oct 29 23:03:25.712999 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712874    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79\" (UniqueName: \"kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79\") pod \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") "
Oct 29 23:03:25.712999 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712925    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs\" (UniqueName: \"kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs\") pod \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\" (UID: \"0976adfc-ecbc-4bb8-94fa-9693dfbcef40\") "
Oct 29 23:03:25.712999 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.712925    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache" pod "0976adfc-ecbc-4bb8-94fa-9693dfbcef40" (UID: "0976adfc-ecbc-4bb8-94fa-9693dfbcef40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.713398 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713100    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home" pod "0976adfc-ecbc-4bb8-94fa-9693dfbcef40" (UID: "0976adfc-ecbc-4bb8-94fa-9693dfbcef40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.713398 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713282    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir" pod "0976adfc-ecbc-4bb8-94fa-9693dfbcef40" (UID: "0976adfc-ecbc-4bb8-94fa-9693dfbcef40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.713550 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713482    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713550 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713504    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713550 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713516    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713550 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713530    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713550 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713547    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnsw7\" (UniqueName: \"kubernetes.io/projected/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-kube-api-access-nnsw7\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713560    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713574    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713587    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmcrp\" (UniqueName: \"kubernetes.io/projected/4c631028-0a42-445c-be49-b8aff3255b33-kube-api-access-cmcrp\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713599    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713612    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c631028-0a42-445c-be49-b8aff3255b33-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713625    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713637    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713648    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdd9e25-f8f5-45f4-aafe-13ac66426e22-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713660    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.713935 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.713673    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c631028-0a42-445c-be49-b8aff3255b33-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.715288 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.715254    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79" pod "0976adfc-ecbc-4bb8-94fa-9693dfbcef40" (UID: "0976adfc-ecbc-4bb8-94fa-9693dfbcef40"). InnerVolumeSpecName "kube-api-access-nhx79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:25.715288 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.715259    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm" pod "0976adfc-ecbc-4bb8-94fa-9693dfbcef40" (UID: "0976adfc-ecbc-4bb8-94fa-9693dfbcef40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.715423 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.715374    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs" pod "0976adfc-ecbc-4bb8-94fa-9693dfbcef40" (UID: "0976adfc-ecbc-4bb8-94fa-9693dfbcef40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:25.815240 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.815157    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhx79\" (UniqueName: \"kubernetes.io/projected/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-kube-api-access-nhx79\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.815240 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.815191    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.815240 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.815205    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0976adfc-ecbc-4bb8-94fa-9693dfbcef40-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.888888 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.888855    4451 generic.go:356] "Generic (PLEG): container finished" podID="4c631028-0a42-445c-be49-b8aff3255b33" containerID="ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6" exitCode=0
Oct 29 23:03:25.889075 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.888930    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"
Oct 29 23:03:25.889075 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.888940    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" event={"ID":"4c631028-0a42-445c-be49-b8aff3255b33","Type":"ContainerDied","Data":"ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6"}
Oct 29 23:03:25.889075 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.888979    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" event={"ID":"4c631028-0a42-445c-be49-b8aff3255b33","Type":"ContainerDied","Data":"5622948663a28da905a432d83df2940bf858f3bc08be3d300631d5c4e3df6939"}
Oct 29 23:03:25.889075 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.889002    4451 scope.go:122] "RemoveContainer" containerID="ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6"
Oct 29 23:03:25.890291 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.890276    4451 generic.go:356] "Generic (PLEG): container finished" podID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerID="fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780" exitCode=0
Oct 29 23:03:25.890382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.890323    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"
Oct 29 23:03:25.890382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.890364    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" event={"ID":"0976adfc-ecbc-4bb8-94fa-9693dfbcef40","Type":"ContainerDied","Data":"fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780"}
Oct 29 23:03:25.890489 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.890409    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" event={"ID":"0976adfc-ecbc-4bb8-94fa-9693dfbcef40","Type":"ContainerDied","Data":"4769c7348a0159730656d3cde1bb46db792f848fd40b13af120d8c97cfca65a9"}
Oct 29 23:03:25.891825 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.891802    4451 generic.go:356] "Generic (PLEG): container finished" podID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerID="a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1" exitCode=0
Oct 29 23:03:25.891965 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.891859    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" event={"ID":"ebdd9e25-f8f5-45f4-aafe-13ac66426e22","Type":"ContainerDied","Data":"a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1"}
Oct 29 23:03:25.891965 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.891881    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"
Oct 29 23:03:25.891965 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.891883    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk" event={"ID":"ebdd9e25-f8f5-45f4-aafe-13ac66426e22","Type":"ContainerDied","Data":"147c405b4f9981f01b0ff724330f8d071e338d249670574fc7b4aee56b99c220"}
Oct 29 23:03:25.900855 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.900819    4451 scope.go:122] "RemoveContainer" containerID="ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6"
Oct 29 23:03:25.901320 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:25.901274    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6\": container with ID starting with ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6 not found: ID does not exist" containerID="ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6"
Oct 29 23:03:25.901480 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.901311    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6"} err="failed to get container status \"ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6\": rpc error: code = NotFound desc = could not find container \"ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6\": container with ID starting with ab4b55f8e836ffc3eb7afcb3d382128d505793783b03617d2966ee1f7be020c6 not found: ID does not exist"
Oct 29 23:03:25.901480 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.901334    4451 scope.go:122] "RemoveContainer" containerID="fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780"
Oct 29 23:03:25.911089 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.911063    4451 scope.go:122] "RemoveContainer" containerID="fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780"
Oct 29 23:03:25.911341 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:25.911323    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780\": container with ID starting with fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780 not found: ID does not exist" containerID="fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780"
Oct 29 23:03:25.911391 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.911346    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780"} err="failed to get container status \"fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780\": rpc error: code = NotFound desc = could not find container \"fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780\": container with ID starting with fe2fdabe4163a8d1b1a228b4c8f57812a40d7c64b4aa947b8646b6b4f5893780 not found: ID does not exist"
Oct 29 23:03:25.911391 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.911361    4451 scope.go:122] "RemoveContainer" containerID="a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1"
Oct 29 23:03:25.919724 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.919701    4451 scope.go:122] "RemoveContainer" containerID="a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1"
Oct 29 23:03:25.920015 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:25.919996    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1\": container with ID starting with a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1 not found: ID does not exist" containerID="a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1"
Oct 29 23:03:25.920099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.920019    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1"} err="failed to get container status \"a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1\": rpc error: code = NotFound desc = could not find container \"a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1\": container with ID starting with a02bb483158faa15e096458522ae9decc26ff1b2c68902d69117300e98d10ad1 not found: ID does not exist"
Oct 29 23:03:25.959440 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.959415    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"]
Oct 29 23:03:25.964661 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.964639    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6"]
Oct 29 23:03:25.993223 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.993195    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"]
Oct 29 23:03:25.997305 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:25.997280    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz"]
Oct 29 23:03:26.028432 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.028406    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"]
Oct 29 23:03:26.031849 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.031817    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/distinct-2-w0-2e046a-kserve-77f85665fd-8lrnk"]
Oct 29 23:03:26.206139 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.206105    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" path="/var/lib/kubelet/pods/0976adfc-ecbc-4bb8-94fa-9693dfbcef40/volumes"
Oct 29 23:03:26.206629 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.206613    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c631028-0a42-445c-be49-b8aff3255b33" path="/var/lib/kubelet/pods/4c631028-0a42-445c-be49-b8aff3255b33/volumes"
Oct 29 23:03:26.207071 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.207055    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" path="/var/lib/kubelet/pods/ebdd9e25-f8f5-45f4-aafe-13ac66426e22/volumes"
Oct 29 23:03:26.459178 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.459090    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/distinct-w0-2e046a-kserve-794b89c4c7-gghk6" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:03:26.535952 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:26.535895    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/premium-w0-2e046a-kserve-6dcff5674b-4q7cz" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 23:03:34.861306 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:34.861268    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main" containerID="cri-o://1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09" gracePeriod=45
Oct 29 23:03:35.125662 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.125598    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 23:03:35.224168 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224135    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs\") pod \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") "
Oct 29 23:03:35.224168 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224173    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir\") pod \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") "
Oct 29 23:03:35.224444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224192    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home\") pod \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") "
Oct 29 23:03:35.224444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224262    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache\") pod \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") "
Oct 29 23:03:35.224444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224295    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm\") pod \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") "
Oct 29 23:03:35.224666 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224494    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir" pod "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" (UID: "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.224666 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224567    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home" pod "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" (UID: "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.224666 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224605    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg\" (UniqueName: \"kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg\") pod \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\" (UID: \"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f\") "
Oct 29 23:03:35.224843 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.224806    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache" pod "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" (UID: "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.225444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.225416    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.225551 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.225447    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.225551 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.225463    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.227289 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.227251    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm" pod "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" (UID: "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.227507 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.227481    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs" pod "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" (UID: "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:35.227611 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.227593    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg" pod "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" (UID: "0b6abeee-43c5-4eb9-a3ac-3bf47515c97f"). InnerVolumeSpecName "kube-api-access-5gdvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:35.326645 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.326603    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.326645 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.326636    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.326645 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.326651    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gdvg\" (UniqueName: \"kubernetes.io/projected/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f-kube-api-access-5gdvg\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.965199 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.965160    4451 generic.go:356] "Generic (PLEG): container finished" podID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerID="1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09" exitCode=0
Oct 29 23:03:35.965199 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.965198    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"
Oct 29 23:03:35.965840 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.965242    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" event={"ID":"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f","Type":"ContainerDied","Data":"1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09"}
Oct 29 23:03:35.965840 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.965287    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf" event={"ID":"0b6abeee-43c5-4eb9-a3ac-3bf47515c97f","Type":"ContainerDied","Data":"0b95280282c57d36a735fc365f211c824a0e8ada0ce0f45ae6684b2b22122c38"}
Oct 29 23:03:35.965840 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.965315    4451 scope.go:122] "RemoveContainer" containerID="1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09"
Oct 29 23:03:35.980581 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.980564    4451 scope.go:122] "RemoveContainer" containerID="1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09"
Oct 29 23:03:35.980836 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:35.980819    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09\": container with ID starting with 1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09 not found: ID does not exist" containerID="1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09"
Oct 29 23:03:35.980887 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:35.980843    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09"} err="failed to get container status \"1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09\": rpc error: code = NotFound desc = could not find container \"1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09\": container with ID starting with 1adb40627e1d426553af3ac4546d6e0c8db5f1dca85ea9c064ba166a122acf09 not found: ID does not exist"
Oct 29 23:03:36.012845 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.012815    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"]
Oct 29 23:03:36.023101 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.023075    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-w2-2ba10a-kserve-6b7557d598-h99gf"]
Oct 29 23:03:36.162422 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.162390    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" path="/var/lib/kubelet/pods/0b6abeee-43c5-4eb9-a3ac-3bf47515c97f/volumes"
Oct 29 23:03:36.523621 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.523572    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"]
Oct 29 23:03:36.523855 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.523825    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" podUID="7ef68324-fbef-4b8a-896e-a5fd11541e4f" containerName="maas-api" containerID="cri-o://2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750" gracePeriod=30
Oct 29 23:03:36.838328 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.838301    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 23:03:36.947724 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.947685    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca\") pod \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") "
Oct 29 23:03:36.947957 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.947800    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\") pod \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") "
Oct 29 23:03:36.947957 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.947849    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs\") pod \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") "
Oct 29 23:03:36.948105 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.947982    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n\" (UniqueName: \"kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n\") pod \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\" (UID: \"7ef68324-fbef-4b8a-896e-a5fd11541e4f\") "
Oct 29 23:03:36.948105 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.948084    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca" pod "7ef68324-fbef-4b8a-896e-a5fd11541e4f" (UID: "7ef68324-fbef-4b8a-896e-a5fd11541e4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:36.948552 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.948533    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef68324-fbef-4b8a-896e-a5fd11541e4f-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:36.950087 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.950060    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls" pod "7ef68324-fbef-4b8a-896e-a5fd11541e4f" (UID: "7ef68324-fbef-4b8a-896e-a5fd11541e4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:36.950208 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.950118    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs" pod "7ef68324-fbef-4b8a-896e-a5fd11541e4f" (UID: "7ef68324-fbef-4b8a-896e-a5fd11541e4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:36.950208 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.950125    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n" pod "7ef68324-fbef-4b8a-896e-a5fd11541e4f" (UID: "7ef68324-fbef-4b8a-896e-a5fd11541e4f"). InnerVolumeSpecName "kube-api-access-2tb9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:36.972673 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.972647    4451 generic.go:356] "Generic (PLEG): container finished" podID="7ef68324-fbef-4b8a-896e-a5fd11541e4f" containerID="2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750" exitCode=0
Oct 29 23:03:36.973052 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.972690    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"
Oct 29 23:03:36.973052 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.972724    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" event={"ID":"7ef68324-fbef-4b8a-896e-a5fd11541e4f","Type":"ContainerDied","Data":"2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750"}
Oct 29 23:03:36.973052 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.972774    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x" event={"ID":"7ef68324-fbef-4b8a-896e-a5fd11541e4f","Type":"ContainerDied","Data":"1af7f558515dec48d97198a9dbfb4af951c1c31f112f95ac67d1dcd8b1a22c20"}
Oct 29 23:03:36.973052 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.972798    4451 scope.go:122] "RemoveContainer" containerID="2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750"
Oct 29 23:03:36.984004 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.983986    4451 scope.go:122] "RemoveContainer" containerID="2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750"
Oct 29 23:03:36.984265 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:36.984247    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750\": container with ID starting with 2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750 not found: ID does not exist" containerID="2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750"
Oct 29 23:03:36.984308 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:36.984274    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750"} err="failed to get container status \"2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750\": rpc error: code = NotFound desc = could not find container \"2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750\": container with ID starting with 2f0b8586b1b1ebcccaa274584b3620e1997eeb8f209e442914736517041b2750 not found: ID does not exist"
Oct 29 23:03:37.050532 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:37.050482    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tb9n\" (UniqueName: \"kubernetes.io/projected/7ef68324-fbef-4b8a-896e-a5fd11541e4f-kube-api-access-2tb9n\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.050532 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:37.050524    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.050532 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:37.050538    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef68324-fbef-4b8a-896e-a5fd11541e4f-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.050844 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:37.050566    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"]
Oct 29 23:03:37.054328 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:37.054148    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-80248b-c469bf89c-9v46x"]
Oct 29 23:03:38.163062 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:38.163029    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef68324-fbef-4b8a-896e-a5fd11541e4f" path="/var/lib/kubelet/pods/7ef68324-fbef-4b8a-896e-a5fd11541e4f/volumes"
Oct 29 23:03:44.037793 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.037291    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"]
Oct 29 23:03:44.037793 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.037527    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg" podUID="c1157cfc-7a7b-4686-a54b-a0942aea1d76" containerName="maas-api" containerID="cri-o://f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6" gracePeriod=30
Oct 29 23:03:44.340320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.340290    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 23:03:44.402901 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.402837    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"]
Oct 29 23:03:44.403842 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.403587    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z" podUID="0e6d20e8-d115-4189-b8b5-219f110d80d1" containerName="maas-api" containerID="cri-o://58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192" gracePeriod=30
Oct 29 23:03:44.433601 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.433557    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs\") pod \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") "
Oct 29 23:03:44.433708 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.433686    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv\" (UniqueName: \"kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv\") pod \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") "
Oct 29 23:03:44.433780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.433765    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca\") pod \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") "
Oct 29 23:03:44.433845 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.433807    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls\") pod \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\" (UID: \"c1157cfc-7a7b-4686-a54b-a0942aea1d76\") "
Oct 29 23:03:44.434648 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.434604    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca" pod "c1157cfc-7a7b-4686-a54b-a0942aea1d76" (UID: "c1157cfc-7a7b-4686-a54b-a0942aea1d76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:44.436669 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.436641    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls" pod "c1157cfc-7a7b-4686-a54b-a0942aea1d76" (UID: "c1157cfc-7a7b-4686-a54b-a0942aea1d76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:44.438180 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.438154    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs" pod "c1157cfc-7a7b-4686-a54b-a0942aea1d76" (UID: "c1157cfc-7a7b-4686-a54b-a0942aea1d76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:44.443266 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.443243    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv" pod "c1157cfc-7a7b-4686-a54b-a0942aea1d76" (UID: "c1157cfc-7a7b-4686-a54b-a0942aea1d76"). InnerVolumeSpecName "kube-api-access-48vmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:44.535340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.535299    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48vmv\" (UniqueName: \"kubernetes.io/projected/c1157cfc-7a7b-4686-a54b-a0942aea1d76-kube-api-access-48vmv\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.535340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.535341    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1157cfc-7a7b-4686-a54b-a0942aea1d76-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.535514 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.535357    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.535514 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.535372    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1157cfc-7a7b-4686-a54b-a0942aea1d76-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.673999 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.673954    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 23:03:44.737872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.736814    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca\") pod \"0e6d20e8-d115-4189-b8b5-219f110d80d1\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") "
Oct 29 23:03:44.737872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.736939    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls\") pod \"0e6d20e8-d115-4189-b8b5-219f110d80d1\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") "
Oct 29 23:03:44.737872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.736985    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs\") pod \"0e6d20e8-d115-4189-b8b5-219f110d80d1\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") "
Oct 29 23:03:44.737872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.737042    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb\" (UniqueName: \"kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb\") pod \"0e6d20e8-d115-4189-b8b5-219f110d80d1\" (UID: \"0e6d20e8-d115-4189-b8b5-219f110d80d1\") "
Oct 29 23:03:44.739797 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.739765    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca" pod "0e6d20e8-d115-4189-b8b5-219f110d80d1" (UID: "0e6d20e8-d115-4189-b8b5-219f110d80d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:44.756988 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.756949    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb" pod "0e6d20e8-d115-4189-b8b5-219f110d80d1" (UID: "0e6d20e8-d115-4189-b8b5-219f110d80d1"). InnerVolumeSpecName "kube-api-access-6kfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:44.757189 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.757080    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs" pod "0e6d20e8-d115-4189-b8b5-219f110d80d1" (UID: "0e6d20e8-d115-4189-b8b5-219f110d80d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:44.758462 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.758434    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls" pod "0e6d20e8-d115-4189-b8b5-219f110d80d1" (UID: "0e6d20e8-d115-4189-b8b5-219f110d80d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:44.838367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.838282    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6d20e8-d115-4189-b8b5-219f110d80d1-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.838367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.838312    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.838367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.838322    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e6d20e8-d115-4189-b8b5-219f110d80d1-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:44.838367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:44.838330    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kfdb\" (UniqueName: \"kubernetes.io/projected/0e6d20e8-d115-4189-b8b5-219f110d80d1-kube-api-access-6kfdb\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.027400 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.027359    4451 generic.go:356] "Generic (PLEG): container finished" podID="0e6d20e8-d115-4189-b8b5-219f110d80d1" containerID="58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192" exitCode=0
Oct 29 23:03:45.027690 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.027419    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"
Oct 29 23:03:45.027690 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.027443    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z" event={"ID":"0e6d20e8-d115-4189-b8b5-219f110d80d1","Type":"ContainerDied","Data":"58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192"}
Oct 29 23:03:45.027690 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.027502    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z" event={"ID":"0e6d20e8-d115-4189-b8b5-219f110d80d1","Type":"ContainerDied","Data":"0acb36b82ba2753168226e240a0c7ba5d7de1680beb942722c1432be5b80cb3f"}
Oct 29 23:03:45.027690 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.027533    4451 scope.go:122] "RemoveContainer" containerID="58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192"
Oct 29 23:03:45.029401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.029315    4451 generic.go:356] "Generic (PLEG): container finished" podID="c1157cfc-7a7b-4686-a54b-a0942aea1d76" containerID="f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6" exitCode=0
Oct 29 23:03:45.029401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.029347    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg" event={"ID":"c1157cfc-7a7b-4686-a54b-a0942aea1d76","Type":"ContainerDied","Data":"f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6"}
Oct 29 23:03:45.029401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.029371    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"
Oct 29 23:03:45.029401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.029391    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg" event={"ID":"c1157cfc-7a7b-4686-a54b-a0942aea1d76","Type":"ContainerDied","Data":"70e2def14598bc73c3cabf1e3629878b428de4826675e694fc2fa2ea157d8c91"}
Oct 29 23:03:45.039566 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.039204    4451 scope.go:122] "RemoveContainer" containerID="58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192"
Oct 29 23:03:45.039566 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:45.039533    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192\": container with ID starting with 58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192 not found: ID does not exist" containerID="58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192"
Oct 29 23:03:45.040020 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.039570    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192"} err="failed to get container status \"58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192\": rpc error: code = NotFound desc = could not find container \"58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192\": container with ID starting with 58efa9b2a603054b096e613ca1c13e81429ccdc2d4e8061a18f2f027c4bde192 not found: ID does not exist"
Oct 29 23:03:45.040020 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.039621    4451 scope.go:122] "RemoveContainer" containerID="f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6"
Oct 29 23:03:45.050149 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.050131    4451 scope.go:122] "RemoveContainer" containerID="f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6"
Oct 29 23:03:45.050413 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:45.050391    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6\": container with ID starting with f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6 not found: ID does not exist" containerID="f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6"
Oct 29 23:03:45.050540 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.050416    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6"} err="failed to get container status \"f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6\": rpc error: code = NotFound desc = could not find container \"f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6\": container with ID starting with f6c31dd8d2cabb8f6f95c4ab3f1fee31ede460161f3c214f4777a7ebd77ad1c6 not found: ID does not exist"
Oct 29 23:03:45.093526 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.093424    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"]
Oct 29 23:03:45.101122 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.101086    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e046a-85fc57fc8d-9hh9z"]
Oct 29 23:03:45.127355 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.127310    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"]
Oct 29 23:03:45.142239 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:45.142204    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-6fc6bd-bc7f57754-vj4sg"]
Oct 29 23:03:46.168371 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:46.168339    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6d20e8-d115-4189-b8b5-219f110d80d1" path="/var/lib/kubelet/pods/0e6d20e8-d115-4189-b8b5-219f110d80d1/volumes"
Oct 29 23:03:46.168979 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:46.168958    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1157cfc-7a7b-4686-a54b-a0942aea1d76" path="/var/lib/kubelet/pods/c1157cfc-7a7b-4686-a54b-a0942aea1d76/volumes"
Oct 29 23:03:46.653249 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:46.653216    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"]
Oct 29 23:03:46.653477 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:46.653443    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" podUID="1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" containerName="maas-api" containerID="cri-o://907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221" gracePeriod=30
Oct 29 23:03:46.968988 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:46.968960    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 23:03:47.071154 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.071119    4451 generic.go:356] "Generic (PLEG): container finished" podID="1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" containerID="907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221" exitCode=0
Oct 29 23:03:47.071339 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.071185    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"
Oct 29 23:03:47.071339 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.071199    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" event={"ID":"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae","Type":"ContainerDied","Data":"907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221"}
Oct 29 23:03:47.071339 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.071259    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79" event={"ID":"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae","Type":"ContainerDied","Data":"36a78f4fcb26d0d1942f78fb1555442462072a81cfdb99c12422f0c5d4315f6a"}
Oct 29 23:03:47.071339 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.071284    4451 scope.go:122] "RemoveContainer" containerID="907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221"
Oct 29 23:03:47.080765 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.080746    4451 scope.go:122] "RemoveContainer" containerID="907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221"
Oct 29 23:03:47.081059 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:47.081037    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221\": container with ID starting with 907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221 not found: ID does not exist" containerID="907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221"
Oct 29 23:03:47.081144 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.081070    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221"} err="failed to get container status \"907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221\": rpc error: code = NotFound desc = could not find container \"907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221\": container with ID starting with 907f8dea4c7dd0490e1ebcb01a3fc580367614624e5f407bb729c20ea4811221 not found: ID does not exist"
Oct 29 23:03:47.087124 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.087106    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\") pod \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") "
Oct 29 23:03:47.087214 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.087151    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs\") pod \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") "
Oct 29 23:03:47.087252 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.087213    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq\" (UniqueName: \"kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq\") pod \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") "
Oct 29 23:03:47.087284 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.087247    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca\") pod \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\" (UID: \"1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae\") "
Oct 29 23:03:47.087635 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.087611    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca" pod "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" (UID: "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:47.087789 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.087771    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:47.089295 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.089269    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs" pod "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" (UID: "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:47.089380 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.089303    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls" pod "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" (UID: "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:47.089436 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.089371    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq" pod "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" (UID: "1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae"). InnerVolumeSpecName "kube-api-access-7xcgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:47.188763 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.188612    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:47.188763 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.188648    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:47.188763 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.188663    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xcgq\" (UniqueName: \"kubernetes.io/projected/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae-kube-api-access-7xcgq\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:47.417632 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.417599    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"]
Oct 29 23:03:47.421539 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:47.421507    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2ba10a-5cf4d5b5d7-5sf79"]
Oct 29 23:03:48.163883 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:48.163828    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" path="/var/lib/kubelet/pods/1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae/volumes"
Oct 29 23:03:48.746111 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:48.745951    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:03:52.117886 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:52.117833    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"]
Oct 29 23:03:52.118396 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:52.118107    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy" containerID="cri-o://f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8" gracePeriod=30
Oct 29 23:03:52.612508 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:52.612471    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"]
Oct 29 23:03:52.613168 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:52.613129    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy" containerID="cri-o://878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b" gracePeriod=30
Oct 29 23:03:54.677452 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:54.677417    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"]
Oct 29 23:03:54.679924 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:54.679857    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy" containerID="cri-o://3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263" gracePeriod=30
Oct 29 23:03:57.414296 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.413893    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 23:03:57.537749 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.537716    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.537933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.537757    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.537933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.537786    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.537933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.537814    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.537933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.537858    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.537933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.537899    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.538223 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538053    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.538223 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538093    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.538223 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538143    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket\") pod \"5ff55386-965c-4a74-9e62-fd0b912ef936\" (UID: \"5ff55386-965c-4a74-9e62-fd0b912ef936\") "
Oct 29 23:03:57.538393 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538260    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:57.538393 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538374    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.538507 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538455    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.538625 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538581    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.538625 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538593    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.538805 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538785    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.538875 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538807    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.538875 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538822    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.538875 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538835    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff55386-965c-4a74-9e62-fd0b912ef936-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.538875 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.538849    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.540243 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.540200    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "kube-api-access-vnrfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.540384 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.540267    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.540654 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.540630    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:57.540719 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.540694    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token" pod "5ff55386-965c-4a74-9e62-fd0b912ef936" (UID: "5ff55386-965c-4a74-9e62-fd0b912ef936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.580982 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.580947    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"]
Oct 29 23:03:57.640023 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.639991    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.640023 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.640022    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrfr\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-kube-api-access-vnrfr\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.640259 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.640037    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.640259 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.640051    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff55386-965c-4a74-9e62-fd0b912ef936-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.893205 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.893170    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 23:03:57.943053 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.942965    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.943671 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.943230    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.943807 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.943709    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.943807 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.943776    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.944276 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.943899    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.944377 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944316    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.944377 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944039    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.944377 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944365    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.944377 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944240    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.945163 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944409    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.945163 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944473    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket\") pod \"f96b660b-d2ed-4dac-8881-a25f40e8913a\" (UID: \"f96b660b-d2ed-4dac-8881-a25f40e8913a\") "
Oct 29 23:03:57.945163 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944970    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.945163 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944979    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:57.945163 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.944993    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.945444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.945203    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.945444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.945283    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.945837 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.945813    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.946149 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.946129    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "kube-api-access-tfnlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.946780 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.946762    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.947320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:57.947304    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo" pod "f96b660b-d2ed-4dac-8881-a25f40e8913a" (UID: "f96b660b-d2ed-4dac-8881-a25f40e8913a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:58.046331 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046295    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnlq\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-kube-api-access-tfnlq\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.046331 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046325    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.046331 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046336    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96b660b-d2ed-4dac-8881-a25f40e8913a-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.046560 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046344    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.046560 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046353    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.046560 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046361    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.046560 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.046369    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96b660b-d2ed-4dac-8881-a25f40e8913a-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.168335 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.168304    4451 generic.go:356] "Generic (PLEG): container finished" podID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerID="878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b" exitCode=0
Oct 29 23:03:58.168537 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.168517    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" event={"ID":"f96b660b-d2ed-4dac-8881-a25f40e8913a","Type":"ContainerDied","Data":"878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b"}
Oct 29 23:03:58.168638 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.168564    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l" event={"ID":"f96b660b-d2ed-4dac-8881-a25f40e8913a","Type":"ContainerDied","Data":"01c3b59cc8630ecf3616fd68e642749493c2ce7e1f322a1e33ea4afec12debe5"}
Oct 29 23:03:58.168638 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.168595    4451 scope.go:122] "RemoveContainer" containerID="878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b"
Oct 29 23:03:58.168755 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.168746    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"
Oct 29 23:03:58.181107 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.178138    4451 generic.go:356] "Generic (PLEG): container finished" podID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerID="f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8" exitCode=0
Oct 29 23:03:58.181107 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.178174    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" event={"ID":"5ff55386-965c-4a74-9e62-fd0b912ef936","Type":"ContainerDied","Data":"f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8"}
Oct 29 23:03:58.181107 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.178340    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"
Oct 29 23:03:58.181107 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.179028    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc" event={"ID":"5ff55386-965c-4a74-9e62-fd0b912ef936","Type":"ContainerDied","Data":"95046d33d7a9a18a2bf3880324cde152c38466cb765a9cd0eca34817e5220e46"}
Oct 29 23:03:58.181490 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.181179    4451 scope.go:122] "RemoveContainer" containerID="878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b"
Oct 29 23:03:58.181635 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:58.181537    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b\": container with ID starting with 878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b not found: ID does not exist" containerID="878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b"
Oct 29 23:03:58.181635 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.181570    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b"} err="failed to get container status \"878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b\": rpc error: code = NotFound desc = could not find container \"878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b\": container with ID starting with 878c1f001a93246419d0ddb25db171622935c93b414c71afb3bcb538494c689b not found: ID does not exist"
Oct 29 23:03:58.181635 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.181594    4451 scope.go:122] "RemoveContainer" containerID="f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8"
Oct 29 23:03:58.190948 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.190927    4451 scope.go:122] "RemoveContainer" containerID="f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8"
Oct 29 23:03:58.191231 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:03:58.191211    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8\": container with ID starting with f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8 not found: ID does not exist" containerID="f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8"
Oct 29 23:03:58.191291 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.191236    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8"} err="failed to get container status \"f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8\": rpc error: code = NotFound desc = could not find container \"f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8\": container with ID starting with f67b2a9ef0cd44d88883efaf4eaccb0ea8db830b38d954df07b6b4ce1f7687e8 not found: ID does not exist"
Oct 29 23:03:58.230445 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.230384    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"]
Oct 29 23:03:58.241571 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.241544    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-lks7l"]
Oct 29 23:03:58.271143 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.271119    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"]
Oct 29 23:03:58.279137 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:58.279115    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-qpsqc"]
Oct 29 23:03:59.951392 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:03:59.951366    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 23:04:00.064969 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.064855    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065128 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.064968    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065128 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065041    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065128 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065082    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065305 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065126    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065305 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065154    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065305 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065168    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.065305 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065209    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065305 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065284    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065546 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065317    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo\") pod \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\" (UID: \"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2\") "
Oct 29 23:04:00.065546 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065342    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.065546 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065432    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.065546 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065440    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.065759 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065738    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.065824 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065764    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.065824 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065780    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.065824 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065795    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.065987 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.065867    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:00.067605 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.067578    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:00.067766 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.067747    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.067851 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.067836    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:00.067895 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.067878    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69" pod "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" (UID: "ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2"). InnerVolumeSpecName "kube-api-access-htr69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:00.164905 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.164870    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" path="/var/lib/kubelet/pods/5ff55386-965c-4a74-9e62-fd0b912ef936/volumes"
Oct 29 23:04:00.165404 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.165387    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" path="/var/lib/kubelet/pods/f96b660b-d2ed-4dac-8881-a25f40e8913a/volumes"
Oct 29 23:04:00.166485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.166458    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.166605 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.166499    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.166605 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.166515    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.166605 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.166530    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htr69\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-kube-api-access-htr69\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.166605 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.166543    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.191241 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.191214    4451 generic.go:356] "Generic (PLEG): container finished" podID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerID="3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263" exitCode=0
Oct 29 23:04:00.191341 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.191249    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" event={"ID":"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2","Type":"ContainerDied","Data":"3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263"}
Oct 29 23:04:00.191341 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.191264    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"
Oct 29 23:04:00.191341 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.191312    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7" event={"ID":"ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2","Type":"ContainerDied","Data":"fc4fbe6c37725ff674932ec54a1b70f6f1bdfc2f82b05868695c29fd21a47593"}
Oct 29 23:04:00.191493 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.191345    4451 scope.go:122] "RemoveContainer" containerID="3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263"
Oct 29 23:04:00.201121 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.201063    4451 scope.go:122] "RemoveContainer" containerID="3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263"
Oct 29 23:04:00.201355 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:00.201335    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263\": container with ID starting with 3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263 not found: ID does not exist" containerID="3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263"
Oct 29 23:04:00.201399 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.201363    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263"} err="failed to get container status \"3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263\": rpc error: code = NotFound desc = could not find container \"3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263\": container with ID starting with 3b4e30afce74f62a1bbc475ff61d0db0ea0137f3fe152b44737042a982279263 not found: ID does not exist"
Oct 29 23:04:00.241011 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.240987    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"]
Oct 29 23:04:00.246847 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:00.246826    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-dccl7"]
Oct 29 23:04:02.162328 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:02.162293    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" path="/var/lib/kubelet/pods/ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2/volumes"
Oct 29 23:04:12.603046 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:12.602984    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main" containerID="cri-o://df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739" gracePeriod=45
Oct 29 23:04:12.859072 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:12.859015    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 23:04:13.018041 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018001    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache\") pod \"5656101a-f6cd-4f3f-884b-f645ea165f35\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") "
Oct 29 23:04:13.018257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018114    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm\") pod \"5656101a-f6cd-4f3f-884b-f645ea165f35\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") "
Oct 29 23:04:13.018257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018160    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf\" (UniqueName: \"kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf\") pod \"5656101a-f6cd-4f3f-884b-f645ea165f35\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") "
Oct 29 23:04:13.018257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018192    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs\" (UniqueName: \"kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs\") pod \"5656101a-f6cd-4f3f-884b-f645ea165f35\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") "
Oct 29 23:04:13.018257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018222    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home\") pod \"5656101a-f6cd-4f3f-884b-f645ea165f35\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") "
Oct 29 23:04:13.018515 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018274    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir\") pod \"5656101a-f6cd-4f3f-884b-f645ea165f35\" (UID: \"5656101a-f6cd-4f3f-884b-f645ea165f35\") "
Oct 29 23:04:13.018515 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018351    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache" pod "5656101a-f6cd-4f3f-884b-f645ea165f35" (UID: "5656101a-f6cd-4f3f-884b-f645ea165f35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:13.018641 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018545    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home" pod "5656101a-f6cd-4f3f-884b-f645ea165f35" (UID: "5656101a-f6cd-4f3f-884b-f645ea165f35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:13.018698 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018652    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir" pod "5656101a-f6cd-4f3f-884b-f645ea165f35" (UID: "5656101a-f6cd-4f3f-884b-f645ea165f35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:13.018817 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018798    4451 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-model-cache\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.018878 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018822    4451 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-home\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.018878 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.018837    4451 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-tmp-dir\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.020427 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.020405    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs" pod "5656101a-f6cd-4f3f-884b-f645ea165f35" (UID: "5656101a-f6cd-4f3f-884b-f645ea165f35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:13.020521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.020478    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm" pod "5656101a-f6cd-4f3f-884b-f645ea165f35" (UID: "5656101a-f6cd-4f3f-884b-f645ea165f35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:13.020521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.020498    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf" pod "5656101a-f6cd-4f3f-884b-f645ea165f35" (UID: "5656101a-f6cd-4f3f-884b-f645ea165f35"). InnerVolumeSpecName "kube-api-access-h55rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:13.120456 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.120390    4451 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5656101a-f6cd-4f3f-884b-f645ea165f35-dshm\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.120456 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.120419    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h55rf\" (UniqueName: \"kubernetes.io/projected/5656101a-f6cd-4f3f-884b-f645ea165f35-kube-api-access-h55rf\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.120456 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.120429    4451 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5656101a-f6cd-4f3f-884b-f645ea165f35-tls-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.265354 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.265311    4451 generic.go:356] "Generic (PLEG): container finished" podID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerID="df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739" exitCode=0
Oct 29 23:04:13.265544 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.265367    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"
Oct 29 23:04:13.265544 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.265372    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" event={"ID":"5656101a-f6cd-4f3f-884b-f645ea165f35","Type":"ContainerDied","Data":"df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739"}
Oct 29 23:04:13.265544 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.265411    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx" event={"ID":"5656101a-f6cd-4f3f-884b-f645ea165f35","Type":"ContainerDied","Data":"104905e141b8b966020b8e45fdf5ae6ab03739bcb262ce23fc44a249ae766a0c"}
Oct 29 23:04:13.265544 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.265453    4451 scope.go:122] "RemoveContainer" containerID="df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739"
Oct 29 23:04:13.275808 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.275790    4451 scope.go:122] "RemoveContainer" containerID="df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739"
Oct 29 23:04:13.276082 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:13.276062    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739\": container with ID starting with df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739 not found: ID does not exist" containerID="df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739"
Oct 29 23:04:13.276146 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.276089    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739"} err="failed to get container status \"df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739\": rpc error: code = NotFound desc = could not find container \"df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739\": container with ID starting with df713d8953a4e23eadcd0eefad90d3c2ab118ceb7b590d748dac37f3f1bf5739 not found: ID does not exist"
Oct 29 23:04:13.310416 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.310389    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"]
Oct 29 23:04:13.314287 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:13.314264    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/premium-w5-22d634-kserve-75d4df9f4-tgpqx"]
Oct 29 23:04:14.163676 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:14.163640    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" path="/var/lib/kubelet/pods/5656101a-f6cd-4f3f-884b-f645ea165f35/volumes"
Oct 29 23:04:18.740987 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:18.740948    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:04:25.742620 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:25.742584    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"]
Oct 29 23:04:25.743469 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:25.743425    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" podUID="7659018a-3ad3-4808-a8bc-857765012d06" containerName="maas-api" containerID="cri-o://4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e" gracePeriod=30
Oct 29 23:04:26.024116 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.024073    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 23:04:26.053540 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.053502    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65\" (UniqueName: \"kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65\") pod \"7659018a-3ad3-4808-a8bc-857765012d06\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") "
Oct 29 23:04:26.053732 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.053655    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca\") pod \"7659018a-3ad3-4808-a8bc-857765012d06\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") "
Oct 29 23:04:26.053732 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.053726    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs\") pod \"7659018a-3ad3-4808-a8bc-857765012d06\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") "
Oct 29 23:04:26.053856 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.053764    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\") pod \"7659018a-3ad3-4808-a8bc-857765012d06\" (UID: \"7659018a-3ad3-4808-a8bc-857765012d06\") "
Oct 29 23:04:26.054187 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.054157    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca" pod "7659018a-3ad3-4808-a8bc-857765012d06" (UID: "7659018a-3ad3-4808-a8bc-857765012d06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:26.054701 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.054674    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7659018a-3ad3-4808-a8bc-857765012d06-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:26.058066 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.058027    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65" pod "7659018a-3ad3-4808-a8bc-857765012d06" (UID: "7659018a-3ad3-4808-a8bc-857765012d06"). InnerVolumeSpecName "kube-api-access-k6c65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:26.059325 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.059298    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs" pod "7659018a-3ad3-4808-a8bc-857765012d06" (UID: "7659018a-3ad3-4808-a8bc-857765012d06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:26.059966 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.059935    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls" pod "7659018a-3ad3-4808-a8bc-857765012d06" (UID: "7659018a-3ad3-4808-a8bc-857765012d06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:26.156003 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.155971    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6c65\" (UniqueName: \"kubernetes.io/projected/7659018a-3ad3-4808-a8bc-857765012d06-kube-api-access-k6c65\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:26.156003 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.156002    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:26.156205 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.156016    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7659018a-3ad3-4808-a8bc-857765012d06-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:26.334203 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.334171    4451 generic.go:356] "Generic (PLEG): container finished" podID="7659018a-3ad3-4808-a8bc-857765012d06" containerID="4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e" exitCode=0
Oct 29 23:04:26.334401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.334245    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" event={"ID":"7659018a-3ad3-4808-a8bc-857765012d06","Type":"ContainerDied","Data":"4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e"}
Oct 29 23:04:26.334401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.334280    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb" event={"ID":"7659018a-3ad3-4808-a8bc-857765012d06","Type":"ContainerDied","Data":"dd1922253406f8dd198e02bca39f0122d4728b39597b09da2eaf7fdb88d6f96b"}
Oct 29 23:04:26.334401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.334300    4451 scope.go:122] "RemoveContainer" containerID="4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e"
Oct 29 23:04:26.334401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.334245    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"
Oct 29 23:04:26.344590 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.344570    4451 scope.go:122] "RemoveContainer" containerID="4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e"
Oct 29 23:04:26.344882 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:26.344854    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e\": container with ID starting with 4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e not found: ID does not exist" containerID="4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e"
Oct 29 23:04:26.344982 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.344894    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e"} err="failed to get container status \"4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e\": rpc error: code = NotFound desc = could not find container \"4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e\": container with ID starting with 4681e21d3e1d64653e5376c5223d22e2607968794d0b34611bee5144bb20a73e not found: ID does not exist"
Oct 29 23:04:26.379637 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.379612    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"]
Oct 29 23:04:26.390165 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:26.390139    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-af260f-6d9b5769-wxltb"]
Oct 29 23:04:28.162673 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:28.162637    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7659018a-3ad3-4808-a8bc-857765012d06" path="/var/lib/kubelet/pods/7659018a-3ad3-4808-a8bc-857765012d06/volumes"
Oct 29 23:04:29.404339 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.404305    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"]
Oct 29 23:04:29.404756 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.404482    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6" podUID="dff77874-4d39-4430-a3a4-f92e49bc33d2" containerName="maas-api" containerID="cri-o://e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658" gracePeriod=30
Oct 29 23:04:29.659581 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.659524    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 23:04:29.693230 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.693198    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs\") pod \"dff77874-4d39-4430-a3a4-f92e49bc33d2\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") "
Oct 29 23:04:29.693358 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.693237    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh\" (UniqueName: \"kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh\") pod \"dff77874-4d39-4430-a3a4-f92e49bc33d2\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") "
Oct 29 23:04:29.693405 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.693369    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca\") pod \"dff77874-4d39-4430-a3a4-f92e49bc33d2\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") "
Oct 29 23:04:29.693442 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.693419    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls\") pod \"dff77874-4d39-4430-a3a4-f92e49bc33d2\" (UID: \"dff77874-4d39-4430-a3a4-f92e49bc33d2\") "
Oct 29 23:04:29.693726 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.693704    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca" pod "dff77874-4d39-4430-a3a4-f92e49bc33d2" (UID: "dff77874-4d39-4430-a3a4-f92e49bc33d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:29.695306 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.695281    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs" pod "dff77874-4d39-4430-a3a4-f92e49bc33d2" (UID: "dff77874-4d39-4430-a3a4-f92e49bc33d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:29.695399 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.695353    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh" pod "dff77874-4d39-4430-a3a4-f92e49bc33d2" (UID: "dff77874-4d39-4430-a3a4-f92e49bc33d2"). InnerVolumeSpecName "kube-api-access-nfzlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:29.695399 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.695383    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls" pod "dff77874-4d39-4430-a3a4-f92e49bc33d2" (UID: "dff77874-4d39-4430-a3a4-f92e49bc33d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:29.794972 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.794899    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:29.794972 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.794964    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfzlh\" (UniqueName: \"kubernetes.io/projected/dff77874-4d39-4430-a3a4-f92e49bc33d2-kube-api-access-nfzlh\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:29.794972 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.794977    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff77874-4d39-4430-a3a4-f92e49bc33d2-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:29.795215 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:29.794986    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff77874-4d39-4430-a3a4-f92e49bc33d2-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:30.356796 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.356758    4451 generic.go:356] "Generic (PLEG): container finished" podID="dff77874-4d39-4430-a3a4-f92e49bc33d2" containerID="e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658" exitCode=0
Oct 29 23:04:30.356796 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.356800    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"
Oct 29 23:04:30.357074 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.356831    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6" event={"ID":"dff77874-4d39-4430-a3a4-f92e49bc33d2","Type":"ContainerDied","Data":"e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658"}
Oct 29 23:04:30.357074 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.356888    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6" event={"ID":"dff77874-4d39-4430-a3a4-f92e49bc33d2","Type":"ContainerDied","Data":"7089aa18d1e8a282f36a2e31288916dfcfc7a0ab696907c6a94727c56b22c24d"}
Oct 29 23:04:30.357074 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.356929    4451 scope.go:122] "RemoveContainer" containerID="e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658"
Oct 29 23:04:30.368194 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.368172    4451 scope.go:122] "RemoveContainer" containerID="e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658"
Oct 29 23:04:30.368488 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:30.368456    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658\": container with ID starting with e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658 not found: ID does not exist" containerID="e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658"
Oct 29 23:04:30.368545 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.368503    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658"} err="failed to get container status \"e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658\": rpc error: code = NotFound desc = could not find container \"e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658\": container with ID starting with e592f94f432e4e95fdd95fe99c2de86f419ec872ac4fc9cf09496fe35633f658 not found: ID does not exist"
Oct 29 23:04:30.411143 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.411111    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"]
Oct 29 23:04:30.424637 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:30.424578    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-22d634-9d8996d9b-k95r6"]
Oct 29 23:04:32.161834 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:32.161804    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff77874-4d39-4430-a3a4-f92e49bc33d2" path="/var/lib/kubelet/pods/dff77874-4d39-4430-a3a4-f92e49bc33d2/volumes"
Oct 29 23:04:33.404121 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:33.403900    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"]
Oct 29 23:04:33.404940 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:33.404895    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy" containerID="cri-o://c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07" gracePeriod=30
Oct 29 23:04:37.362600 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:37.362560    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"]
Oct 29 23:04:37.363039 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:37.362889    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy" containerID="cri-o://d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b" gracePeriod=30
Oct 29 23:04:38.665125 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.665093    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 23:04:38.789801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.789716    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.789801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.789768    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.789801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.789796    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790156 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.789828    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790156 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.789892    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790156 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.789972    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790156 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790012    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790156 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790051    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790156 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790119    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket\") pod \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\" (UID: \"3a29d063-6510-4fe3-8d38-1c24b3a5c70b\") "
Oct 29 23:04:38.790547 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790517    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:38.790547 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790528    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.790801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790682    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.790801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790742    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.790801 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.790777    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.791257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.791232    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.791367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.791294    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.791367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.791313    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.791367 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.791327    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.792622 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.792597    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.792753 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.792730    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:38.793064 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.793045    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "kube-api-access-x77m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.793164 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.793137    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy" pod "3a29d063-6510-4fe3-8d38-1c24b3a5c70b" (UID: "3a29d063-6510-4fe3-8d38-1c24b3a5c70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.892818 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.892782    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.892818 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.892813    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.892818 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.892822    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.893093 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.892830    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x77m4\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-kube-api-access-x77m4\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.893093 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:38.892840    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a29d063-6510-4fe3-8d38-1c24b3a5c70b-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:39.426722 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.426679    4451 generic.go:356] "Generic (PLEG): container finished" podID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerID="c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07" exitCode=0
Oct 29 23:04:39.426899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.426723    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" event={"ID":"3a29d063-6510-4fe3-8d38-1c24b3a5c70b","Type":"ContainerDied","Data":"c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07"}
Oct 29 23:04:39.426899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.426756    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"
Oct 29 23:04:39.426899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.426827    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz" event={"ID":"3a29d063-6510-4fe3-8d38-1c24b3a5c70b","Type":"ContainerDied","Data":"24724aff10d29988c571a3fff4f98ddee9c27a3ed2f9d66eff035553c6efbfd1"}
Oct 29 23:04:39.426899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.426861    4451 scope.go:122] "RemoveContainer" containerID="c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07"
Oct 29 23:04:39.437068 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.437045    4451 scope.go:122] "RemoveContainer" containerID="c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07"
Oct 29 23:04:39.437345 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:39.437324    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07\": container with ID starting with c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07 not found: ID does not exist" containerID="c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07"
Oct 29 23:04:39.437411 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.437354    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07"} err="failed to get container status \"c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07\": rpc error: code = NotFound desc = could not find container \"c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07\": container with ID starting with c4c6e73dade95a217aea7438a3b65048b174d934a86b7e9690e55c01f959fa07 not found: ID does not exist"
Oct 29 23:04:39.482304 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.482271    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"]
Oct 29 23:04:39.492671 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:39.492632    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wkhfz"]
Oct 29 23:04:40.163600 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:40.163569    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" path="/var/lib/kubelet/pods/3a29d063-6510-4fe3-8d38-1c24b3a5c70b/volumes"
Oct 29 23:04:42.624549 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.624521    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 23:04:42.737054 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.736979    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737054 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737032    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737063    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737086    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737114    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737169    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737193    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737226    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737274 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737254    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token\") pod \"945fa94d-f6f9-4270-b17d-3b40c04860c4\" (UID: \"945fa94d-f6f9-4270-b17d-3b40c04860c4\") "
Oct 29 23:04:42.737644 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737497    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:42.737644 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737511    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.737644 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737528    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.737644 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737591    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.737857 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737694    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.737857 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737825    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.737857 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737847    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945fa94d-f6f9-4270-b17d-3b40c04860c4-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.738046 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737862    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.738046 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737877    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.738046 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.737890    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.739426 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.739403    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "kube-api-access-sm25v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:42.739636 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.739619    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:42.739772 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.739756    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:42.739820 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.739799    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy" pod "945fa94d-f6f9-4270-b17d-3b40c04860c4" (UID: "945fa94d-f6f9-4270-b17d-3b40c04860c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.838738 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.838700    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm25v\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-kube-api-access-sm25v\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.838738 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.838729    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.838738 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.838739    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.838738 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:42.838748    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945fa94d-f6f9-4270-b17d-3b40c04860c4-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:43.447739 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.447699    4451 generic.go:356] "Generic (PLEG): container finished" podID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerID="d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b" exitCode=0
Oct 29 23:04:43.447944 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.447759    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"
Oct 29 23:04:43.447944 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.447788    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" event={"ID":"945fa94d-f6f9-4270-b17d-3b40c04860c4","Type":"ContainerDied","Data":"d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b"}
Oct 29 23:04:43.447944 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.447850    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j" event={"ID":"945fa94d-f6f9-4270-b17d-3b40c04860c4","Type":"ContainerDied","Data":"b67e9ed704aee975788c4fdb18ed88c2cd64ed0bb449c37b323e0fcb31ee19c4"}
Oct 29 23:04:43.447944 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.447879    4451 scope.go:122] "RemoveContainer" containerID="d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b"
Oct 29 23:04:43.458202 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.458187    4451 scope.go:122] "RemoveContainer" containerID="d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b"
Oct 29 23:04:43.458465 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:43.458441    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b\": container with ID starting with d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b not found: ID does not exist" containerID="d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b"
Oct 29 23:04:43.458535 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.458478    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b"} err="failed to get container status \"d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b\": rpc error: code = NotFound desc = could not find container \"d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b\": container with ID starting with d6cb47870148839a36edeff2d7243f4e43b12062966b5b57470cd6abc4ab4d4b not found: ID does not exist"
Oct 29 23:04:43.509811 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.509777    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"]
Oct 29 23:04:43.520477 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:43.520445    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-nbn5j"]
Oct 29 23:04:44.161819 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:44.161779    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" path="/var/lib/kubelet/pods/945fa94d-f6f9-4270-b17d-3b40c04860c4/volumes"
Oct 29 23:04:48.741570 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:48.741532    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:04:54.641942 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:54.641893    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"]
Oct 29 23:04:54.642603 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:54.642136    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy" containerID="cri-o://d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3" gracePeriod=30
Oct 29 23:04:54.651902 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:54.651867    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"]
Oct 29 23:04:54.652599 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:54.652568    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy" containerID="cri-o://0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259" gracePeriod=30
Oct 29 23:04:58.012928 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.012883    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"]
Oct 29 23:04:58.013340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.013113    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" podUID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" containerName="discovery" containerID="cri-o://52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a" gracePeriod=30
Oct 29 23:04:58.270167 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.270107    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 23:04:58.293172 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293144    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293300 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293205    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293300 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293266    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293479 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293346    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293479 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293380    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293479 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293434    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293638 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293514    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn\") pod \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\" (UID: \"3f36b40c-e687-4e2f-a34b-1d483b80ead5\") "
Oct 29 23:04:58.293690 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293670    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:58.293881 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.293857    4451 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-ca-configmap\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.295659 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.295636    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:58.295790 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.295771    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "kube-api-access-hggmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:58.295844 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.295810    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:58.295844 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.295814    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:58.296064 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.296044    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:58.296299 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.296283    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig" pod "3f36b40c-e687-4e2f-a34b-1d483b80ead5" (UID: "3f36b40c-e687-4e2f-a34b-1d483b80ead5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:58.394925 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.394880    4451 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-csr-dns-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.395084 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.394935    4451 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-cacerts\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.395084 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.394953    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggmn\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-kube-api-access-hggmn\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.395084 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.394966    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.395084 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.394978    4451 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f36b40c-e687-4e2f-a34b-1d483b80ead5-local-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.395084 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.394988    4451 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f36b40c-e687-4e2f-a34b-1d483b80ead5-istio-kubeconfig\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.538668 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.538584    4451 generic.go:356] "Generic (PLEG): container finished" podID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" containerID="52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a" exitCode=0
Oct 29 23:04:58.538668 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.538633    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"
Oct 29 23:04:58.538861 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.538673    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" event={"ID":"3f36b40c-e687-4e2f-a34b-1d483b80ead5","Type":"ContainerDied","Data":"52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a"}
Oct 29 23:04:58.538861 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.538724    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89" event={"ID":"3f36b40c-e687-4e2f-a34b-1d483b80ead5","Type":"ContainerDied","Data":"14e74e6728a3aa165a454496e0667542f617de16298e02c55d5933bdd2c18da6"}
Oct 29 23:04:58.538861 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.538744    4451 scope.go:122] "RemoveContainer" containerID="52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a"
Oct 29 23:04:58.548833 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.548815    4451 scope.go:122] "RemoveContainer" containerID="52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a"
Oct 29 23:04:58.549133 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:04:58.549114    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a\": container with ID starting with 52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a not found: ID does not exist" containerID="52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a"
Oct 29 23:04:58.549202 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.549146    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a"} err="failed to get container status \"52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a\": rpc error: code = NotFound desc = could not find container \"52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a\": container with ID starting with 52ad6852bd3439d55f6279dd9185532820d88aaf2909ee77a557db0b13bd481a not found: ID does not exist"
Oct 29 23:04:58.584381 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.584349    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"]
Oct 29 23:04:58.590731 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:58.590708    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9jn89"]
Oct 29 23:04:59.895079 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.895050    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 23:04:59.909987 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.909931    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910094 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.909995    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910094 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910030    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910094 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910053    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910094 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910077    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910106    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910167    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910195    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910241    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token\") pod \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\" (UID: \"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1\") "
Oct 29 23:04:59.910361 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910305    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.910651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910419    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.910651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910466    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.910651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910498    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:59.910809 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910684    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.910809 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910749    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.910809 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910776    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.910809 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910790    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.910809 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.910803    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.912619 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.912597    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "kube-api-access-66s9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:59.912700 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.912672    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:59.912872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.912852    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.912872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.912860    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo" pod "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" (UID: "b9d30dc2-8472-42f0-ab7c-e4c05bc463f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:59.930873 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:04:59.930852    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 23:05:00.011702 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.011666    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.011892 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.011720    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.011892 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.011759    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.011892 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.011837    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.012099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.011895    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.012099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.011955    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.012099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012028    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.012099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012037    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.012099 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012071    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.012301 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012109    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token\") pod \"392db7df-d6ab-4577-93fe-358d4f076821\" (UID: \"392db7df-d6ab-4577-93fe-358d4f076821\") "
Oct 29 23:05:00.012301 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012116    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.012386 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012345    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.012523 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012502    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012593 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012528    4451 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-credential-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012593 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012543    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66s9n\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-kube-api-access-66s9n\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012593 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012556    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012593 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012570    4451 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-socket\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012593 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012581    4451 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-data\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012854 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012588    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.012854 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012594    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012854 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012620    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.012854 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.012773    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:00.014142 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.014118    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.014255 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.014239    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.014325 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.014307    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "kube-api-access-fkt2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.014442 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.014424    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo" pod "392db7df-d6ab-4577-93fe-358d4f076821" (UID: "392db7df-d6ab-4577-93fe-358d4f076821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:00.113337 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.113305    4451 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392db7df-d6ab-4577-93fe-358d4f076821-istio-podinfo\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.113337 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.113335    4451 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392db7df-d6ab-4577-93fe-358d4f076821-istiod-ca-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.113521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.113347    4451 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-istio-envoy\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.113521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.113359    4451 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392db7df-d6ab-4577-93fe-358d4f076821-workload-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.113521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.113370    4451 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-istio-token\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.113521 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.113381    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkt2n\" (UniqueName: \"kubernetes.io/projected/392db7df-d6ab-4577-93fe-358d4f076821-kube-api-access-fkt2n\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.165137 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.165063    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" path="/var/lib/kubelet/pods/3f36b40c-e687-4e2f-a34b-1d483b80ead5/volumes"
Oct 29 23:05:00.551518 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.551421    4451 generic.go:356] "Generic (PLEG): container finished" podID="392db7df-d6ab-4577-93fe-358d4f076821" containerID="0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259" exitCode=0
Oct 29 23:05:00.551518 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.551497    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"
Oct 29 23:05:00.551747 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.551552    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" event={"ID":"392db7df-d6ab-4577-93fe-358d4f076821","Type":"ContainerDied","Data":"0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259"}
Oct 29 23:05:00.551747 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.551623    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr" event={"ID":"392db7df-d6ab-4577-93fe-358d4f076821","Type":"ContainerDied","Data":"9acaa696414273443438dfb5f4cb75ad892ad4dfe3b09515b9c5a7ab2b7ca4c1"}
Oct 29 23:05:00.551747 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.551651    4451 scope.go:122] "RemoveContainer" containerID="0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259"
Oct 29 23:05:00.554986 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.554960    4451 generic.go:356] "Generic (PLEG): container finished" podID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerID="d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3" exitCode=0
Oct 29 23:05:00.555125 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.555068    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" event={"ID":"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1","Type":"ContainerDied","Data":"d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3"}
Oct 29 23:05:00.555125 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.555108    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2" event={"ID":"b9d30dc2-8472-42f0-ab7c-e4c05bc463f1","Type":"ContainerDied","Data":"ba569fcecdcc4417c675cd1721957af5d750051c50edc190626c17b2d2795900"}
Oct 29 23:05:00.555253 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.555190    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"
Oct 29 23:05:00.562543 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.562515    4451 scope.go:122] "RemoveContainer" containerID="0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259"
Oct 29 23:05:00.562808 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:05:00.562788    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259\": container with ID starting with 0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259 not found: ID does not exist" containerID="0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259"
Oct 29 23:05:00.562872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.562815    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259"} err="failed to get container status \"0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259\": rpc error: code = NotFound desc = could not find container \"0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259\": container with ID starting with 0dc42c62deb1345294744e2b9354db77c01b2543a180104cb8ec4d07282cc259 not found: ID does not exist"
Oct 29 23:05:00.562872 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.562834    4451 scope.go:122] "RemoveContainer" containerID="d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3"
Oct 29 23:05:00.572010 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.571989    4451 scope.go:122] "RemoveContainer" containerID="d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3"
Oct 29 23:05:00.572282 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:05:00.572265    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3\": container with ID starting with d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3 not found: ID does not exist" containerID="d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3"
Oct 29 23:05:00.572339 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.572288    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3"} err="failed to get container status \"d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3\": rpc error: code = NotFound desc = could not find container \"d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3\": container with ID starting with d258fdca4dd34cc19f8669dfd8a15da7337a1c4e1c0cb96865354001197ec1e3 not found: ID does not exist"
Oct 29 23:05:00.608314 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.608276    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"]
Oct 29 23:05:00.618803 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.618774    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-wrrbr"]
Oct 29 23:05:00.646450 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.646421    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"]
Oct 29 23:05:00.650882 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:00.650856    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-n69t2"]
Oct 29 23:05:02.161746 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:02.161707    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392db7df-d6ab-4577-93fe-358d4f076821" path="/var/lib/kubelet/pods/392db7df-d6ab-4577-93fe-358d4f076821/volumes"
Oct 29 23:05:02.162211 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:02.162195    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" path="/var/lib/kubelet/pods/b9d30dc2-8472-42f0-ab7c-e4c05bc463f1/volumes"
Oct 29 23:05:12.030579 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:12.030543    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"]
Oct 29 23:05:12.040792 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:12.040764    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b23552-nkh9f"]
Oct 29 23:05:12.161350 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:12.161317    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9792b127-6a0c-4287-ab5d-70bcd54a211f" path="/var/lib/kubelet/pods/9792b127-6a0c-4287-ab5d-70bcd54a211f/volumes"
Oct 29 23:05:18.740831 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:05:18.740792    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:05:23.580308 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.580271    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"]
Oct 29 23:05:23.580802 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.580564    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" podUID="6670afdc-0e6d-485e-9f49-5b8578979c1e" containerName="maas-api" containerID="cri-o://18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba" gracePeriod=30
Oct 29 23:05:23.829077 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.829053    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 23:05:23.933103 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.933012    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs\") pod \"6670afdc-0e6d-485e-9f49-5b8578979c1e\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") "
Oct 29 23:05:23.933103 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.933081    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca\") pod \"6670afdc-0e6d-485e-9f49-5b8578979c1e\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") "
Oct 29 23:05:23.933344 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.933188    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw\" (UniqueName: \"kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw\") pod \"6670afdc-0e6d-485e-9f49-5b8578979c1e\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") "
Oct 29 23:05:23.933344 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.933236    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\") pod \"6670afdc-0e6d-485e-9f49-5b8578979c1e\" (UID: \"6670afdc-0e6d-485e-9f49-5b8578979c1e\") "
Oct 29 23:05:23.933535 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.933508    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca" pod "6670afdc-0e6d-485e-9f49-5b8578979c1e" (UID: "6670afdc-0e6d-485e-9f49-5b8578979c1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:23.935404 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.935379    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw" pod "6670afdc-0e6d-485e-9f49-5b8578979c1e" (UID: "6670afdc-0e6d-485e-9f49-5b8578979c1e"). InnerVolumeSpecName "kube-api-access-rb9zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:23.935404 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.935392    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs" pod "6670afdc-0e6d-485e-9f49-5b8578979c1e" (UID: "6670afdc-0e6d-485e-9f49-5b8578979c1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:23.935596 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:23.935457    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls" pod "6670afdc-0e6d-485e-9f49-5b8578979c1e" (UID: "6670afdc-0e6d-485e-9f49-5b8578979c1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:24.034386 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.034351    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:24.034386 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.034383    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6670afdc-0e6d-485e-9f49-5b8578979c1e-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:24.034386 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.034394    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9zw\" (UniqueName: \"kubernetes.io/projected/6670afdc-0e6d-485e-9f49-5b8578979c1e-kube-api-access-rb9zw\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:24.034618 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.034405    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6670afdc-0e6d-485e-9f49-5b8578979c1e-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:24.672024 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.671991    4451 generic.go:356] "Generic (PLEG): container finished" podID="6670afdc-0e6d-485e-9f49-5b8578979c1e" containerID="18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba" exitCode=0
Oct 29 23:05:24.672461 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.672035    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"
Oct 29 23:05:24.672461 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.672059    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" event={"ID":"6670afdc-0e6d-485e-9f49-5b8578979c1e","Type":"ContainerDied","Data":"18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba"}
Oct 29 23:05:24.672461 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.672105    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb" event={"ID":"6670afdc-0e6d-485e-9f49-5b8578979c1e","Type":"ContainerDied","Data":"c4d10b6db966753fe994109eedcb3bc852682c149e5eb0e42767ba580659f412"}
Oct 29 23:05:24.672461 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.672126    4451 scope.go:122] "RemoveContainer" containerID="18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba"
Oct 29 23:05:24.681315 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.681296    4451 scope.go:122] "RemoveContainer" containerID="18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba"
Oct 29 23:05:24.681574 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:05:24.681552    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba\": container with ID starting with 18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba not found: ID does not exist" containerID="18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba"
Oct 29 23:05:24.681624 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.681583    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba"} err="failed to get container status \"18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba\": rpc error: code = NotFound desc = could not find container \"18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba\": container with ID starting with 18b063f917339eefe84502b6049199e3e54680f683d515cd8650fbd6fc866bba not found: ID does not exist"
Oct 29 23:05:24.717085 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.717059    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"]
Oct 29 23:05:24.727169 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:24.727143    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fc684b-77b669878b-mxlgb"]
Oct 29 23:05:26.162011 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:26.161981    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6670afdc-0e6d-485e-9f49-5b8578979c1e" path="/var/lib/kubelet/pods/6670afdc-0e6d-485e-9f49-5b8578979c1e/volumes"
Oct 29 23:05:33.422553 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.422510    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-g9wkx"]
Oct 29 23:05:33.423025 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.422718    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx" podUID="6f0af4f9-4d1d-439b-b18f-723efd65ec59" containerName="manager" containerID="cri-o://df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9" gracePeriod=30
Oct 29 23:05:33.658371 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.658343    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 23:05:33.714836 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.714754    4451 generic.go:356] "Generic (PLEG): container finished" podID="6f0af4f9-4d1d-439b-b18f-723efd65ec59" containerID="df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9" exitCode=0
Oct 29 23:05:33.714836 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.714796    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx" event={"ID":"6f0af4f9-4d1d-439b-b18f-723efd65ec59","Type":"ContainerDied","Data":"df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9"}
Oct 29 23:05:33.714836 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.714808    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx"
Oct 29 23:05:33.714836 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.714836    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-g9wkx" event={"ID":"6f0af4f9-4d1d-439b-b18f-723efd65ec59","Type":"ContainerDied","Data":"e37716798b98541a156907bbc496fba331942587fcdb3c021983704b8d8ee145"}
Oct 29 23:05:33.715183 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.714857    4451 scope.go:122] "RemoveContainer" containerID="df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9"
Oct 29 23:05:33.727829 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.727805    4451 scope.go:122] "RemoveContainer" containerID="df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9"
Oct 29 23:05:33.728173 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:05:33.728154    4451 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9\": container with ID starting with df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9 not found: ID does not exist" containerID="df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9"
Oct 29 23:05:33.728248 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.728183    4451 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9"} err="failed to get container status \"df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9\": rpc error: code = NotFound desc = could not find container \"df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9\": container with ID starting with df0c5397dded04f9de4ae4aa0041d63475f54ecd6d49dff6341f326cc8c0dab9 not found: ID does not exist"
Oct 29 23:05:33.821639 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.821610    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert\") pod \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") "
Oct 29 23:05:33.821825 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.821708    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs\") pod \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") "
Oct 29 23:05:33.821825 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.821751    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8\" (UniqueName: \"kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8\") pod \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\" (UID: \"6f0af4f9-4d1d-439b-b18f-723efd65ec59\") "
Oct 29 23:05:33.823837 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.823805    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert" pod "6f0af4f9-4d1d-439b-b18f-723efd65ec59" (UID: "6f0af4f9-4d1d-439b-b18f-723efd65ec59"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:33.823960 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.823856    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs" pod "6f0af4f9-4d1d-439b-b18f-723efd65ec59" (UID: "6f0af4f9-4d1d-439b-b18f-723efd65ec59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:33.823960 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.823876    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8" pod "6f0af4f9-4d1d-439b-b18f-723efd65ec59" (UID: "6f0af4f9-4d1d-439b-b18f-723efd65ec59"). InnerVolumeSpecName "kube-api-access-22gg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:33.922487 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.922453    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:33.922487 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.922483    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gg8\" (UniqueName: \"kubernetes.io/projected/6f0af4f9-4d1d-439b-b18f-723efd65ec59-kube-api-access-22gg8\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:33.922487 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:33.922496    4451 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f0af4f9-4d1d-439b-b18f-723efd65ec59-webhook-cert\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:05:34.056051 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:34.056022    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-g9wkx"]
Oct 29 23:05:34.059106 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:34.059081    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-g9wkx"]
Oct 29 23:05:34.161867 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:05:34.161840    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0af4f9-4d1d-439b-b18f-723efd65ec59" path="/var/lib/kubelet/pods/6f0af4f9-4d1d-439b-b18f-723efd65ec59/volumes"
Oct 29 23:05:48.740411 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:05:48.740367    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:06:18.739789 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:06:18.739752    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:06:19.595032 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.594995    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"]
Oct 29 23:06:19.595400 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595385    4451 cpu_manager.go:414] "containerMap: removing container" podUID="dff77874-4d39-4430-a3a4-f92e49bc33d2" containerName="maas-api"
Oct 29 23:06:19.595400 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595401    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff77874-4d39-4430-a3a4-f92e49bc33d2" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595418    4451 cpu_manager.go:414] "containerMap: removing container" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595428    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595442    4451 cpu_manager.go:414] "containerMap: removing container" podUID="fbf17621-84f3-4c2c-a7a6-b41679a95a08" containerName="main"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595451    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf17621-84f3-4c2c-a7a6-b41679a95a08" containerName="main"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595458    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7ef68324-fbef-4b8a-896e-a5fd11541e4f" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595463    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef68324-fbef-4b8a-896e-a5fd11541e4f" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595475    4451 cpu_manager.go:414] "containerMap: removing container" podUID="d111483e-be3a-4478-bbfa-c55215de5da4" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595480    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d111483e-be3a-4478-bbfa-c55215de5da4" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595488    4451 cpu_manager.go:414] "containerMap: removing container" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595492    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595499    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595503    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595511    4451 cpu_manager.go:414] "containerMap: removing container" podUID="0e6d20e8-d115-4189-b8b5-219f110d80d1" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595515    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6d20e8-d115-4189-b8b5-219f110d80d1" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595520    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595524    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595531    4451 cpu_manager.go:414] "containerMap: removing container" podUID="193bef37-cee5-4696-967d-db3ad0bd76c8" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595536    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193bef37-cee5-4696-967d-db3ad0bd76c8" containerName="maas-api"
Oct 29 23:06:19.595538 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595543    4451 cpu_manager.go:414] "containerMap: removing container" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595548    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595557    4451 cpu_manager.go:414] "containerMap: removing container" podUID="be906733-a9c7-4738-99ee-40d41b4824a2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595561    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be906733-a9c7-4738-99ee-40d41b4824a2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595567    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595571    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595579    4451 cpu_manager.go:414] "containerMap: removing container" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595584    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595591    4451 cpu_manager.go:414] "containerMap: removing container" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595595    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595601    4451 cpu_manager.go:414] "containerMap: removing container" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595605    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595611    4451 cpu_manager.go:414] "containerMap: removing container" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595615    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595622    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6f0af4f9-4d1d-439b-b18f-723efd65ec59" containerName="manager"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595626    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0af4f9-4d1d-439b-b18f-723efd65ec59" containerName="manager"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595631    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595636    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595644    4451 cpu_manager.go:414] "containerMap: removing container" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595648    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595655    4451 cpu_manager.go:414] "containerMap: removing container" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595659    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595666    4451 cpu_manager.go:414] "containerMap: removing container" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595671    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595676    4451 cpu_manager.go:414] "containerMap: removing container" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595680    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595685    4451 cpu_manager.go:414] "containerMap: removing container" podUID="41d9ebb5-d732-4eb5-86a7-f088a59bfd04" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595689    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d9ebb5-d732-4eb5-86a7-f088a59bfd04" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595694    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595698    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595704    4451 cpu_manager.go:414] "containerMap: removing container" podUID="9792b127-6a0c-4287-ab5d-70bcd54a211f" containerName="revoke-keys"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595708    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9792b127-6a0c-4287-ab5d-70bcd54a211f" containerName="revoke-keys"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595715    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c1157cfc-7a7b-4686-a54b-a0942aea1d76" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595719    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1157cfc-7a7b-4686-a54b-a0942aea1d76" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595725    4451 cpu_manager.go:414] "containerMap: removing container" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595729    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595738    4451 cpu_manager.go:414] "containerMap: removing container" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595743    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595748    4451 cpu_manager.go:414] "containerMap: removing container" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595753    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595759    4451 cpu_manager.go:414] "containerMap: removing container" podUID="770b4385-960b-4f68-9532-483d0dae622c" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595763    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770b4385-960b-4f68-9532-483d0dae622c" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595769    4451 cpu_manager.go:414] "containerMap: removing container" podUID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" containerName="discovery"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595773    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" containerName="discovery"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595781    4451 cpu_manager.go:414] "containerMap: removing container" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595785    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595791    4451 cpu_manager.go:414] "containerMap: removing container" podUID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595795    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595802    4451 cpu_manager.go:414] "containerMap: removing container" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595806    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595811    4451 cpu_manager.go:414] "containerMap: removing container" podUID="fcad3545-c689-4336-8fba-719bdd911ae3" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595815    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcad3545-c689-4336-8fba-719bdd911ae3" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595821    4451 cpu_manager.go:414] "containerMap: removing container" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595825    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595832    4451 cpu_manager.go:414] "containerMap: removing container" podUID="7659018a-3ad3-4808-a8bc-857765012d06" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595838    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7659018a-3ad3-4808-a8bc-857765012d06" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595847    4451 cpu_manager.go:414] "containerMap: removing container" podUID="a038013b-5758-420e-a8f0-05ba413a1fcb" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595851    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a038013b-5758-420e-a8f0-05ba413a1fcb" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595858    4451 cpu_manager.go:414] "containerMap: removing container" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595862    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595870    4451 cpu_manager.go:414] "containerMap: removing container" podUID="1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595874    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595883    4451 cpu_manager.go:414] "containerMap: removing container" podUID="6670afdc-0e6d-485e-9f49-5b8578979c1e" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595887    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6670afdc-0e6d-485e-9f49-5b8578979c1e" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595894    4451 cpu_manager.go:414] "containerMap: removing container" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595898    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595904    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595921    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595978    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff36cacd-fec0-41a2-aa34-d05f20689ffc" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595988    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="9792b127-6a0c-4287-ab5d-70bcd54a211f" containerName="revoke-keys"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.595995    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1157cfc-7a7b-4686-a54b-a0942aea1d76" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596001    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0959bf2-4b14-4956-a695-fddffd81a9aa" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596008    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5c8db1-3048-414d-822b-2aa15c83932b" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596012    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6abeee-43c5-4eb9-a3ac-3bf47515c97f" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596019    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f36b40c-e687-4e2f-a34b-1d483b80ead5" containerName="discovery"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596024    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="770b4385-960b-4f68-9532-483d0dae622c" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596030    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="945fa94d-f6f9-4270-b17d-3b40c04860c4" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596035    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="1851e1e9-8320-40d3-92fe-52d8e8d62ef2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596040    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecac9340-7c1f-4a3c-9dfb-36a0c375d1e2" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596046    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcad3545-c689-4336-8fba-719bdd911ae3" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596052    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ccc1a0-98cb-4366-bb9b-6bca93a999a1" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596058    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7659018a-3ad3-4808-a8bc-857765012d06" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596064    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a038013b-5758-420e-a8f0-05ba413a1fcb" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596068    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb53b1d8-2f2b-41c9-88f4-58ca148122be" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596073    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef5b0a3-a1ae-43cc-8a3e-c4f06eb903ae" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596078    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6670afdc-0e6d-485e-9f49-5b8578979c1e" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596083    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="5656101a-f6cd-4f3f-884b-f645ea165f35" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596088    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c316150f-802e-4b9e-97eb-2611bfdf2ae3" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596094    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff77874-4d39-4430-a3a4-f92e49bc33d2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596099    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d30dc2-8472-42f0-ab7c-e4c05bc463f1" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596104    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf17621-84f3-4c2c-a7a6-b41679a95a08" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596108    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef68324-fbef-4b8a-896e-a5fd11541e4f" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596114    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="d111483e-be3a-4478-bbfa-c55215de5da4" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596119    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="0976adfc-ecbc-4bb8-94fa-9693dfbcef40" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596123    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75d478e-e111-427c-b820-a5a33314e693" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596128    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6d20e8-d115-4189-b8b5-219f110d80d1" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596134    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdd9e25-f8f5-45f4-aafe-13ac66426e22" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596139    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="193bef37-cee5-4696-967d-db3ad0bd76c8" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596144    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="5130665c-f681-4734-9356-0046af52ee3c" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596150    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="be906733-a9c7-4738-99ee-40d41b4824a2" containerName="maas-api"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596154    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="7598412d-8b77-48fb-8d20-ee60f736228e" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596160    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff55386-965c-4a74-9e62-fd0b912ef936" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596164    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a8d410-df76-4882-9764-3227df225944" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596170    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="392db7df-d6ab-4577-93fe-358d4f076821" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596175    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a29d063-6510-4fe3-8d38-1c24b3a5c70b" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596180    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0af4f9-4d1d-439b-b18f-723efd65ec59" containerName="manager"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596185    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96b660b-d2ed-4dac-8881-a25f40e8913a" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596190    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21a6e4e-b9a1-4c04-b0a6-1d60591b19bc" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596194    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6a29c6-375d-40d5-a2f2-144bc504e51f" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596199    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c631028-0a42-445c-be49-b8aff3255b33" containerName="main"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596204    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e810219-babe-4d39-9b85-66d07fd14df6" containerName="istio-proxy"
Oct 29 23:06:19.596204 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.596209    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d9ebb5-d732-4eb5-86a7-f088a59bfd04" containerName="maas-api"
Oct 29 23:06:19.599582 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.599400    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.624981 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.624953    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"]
Oct 29 23:06:19.744018 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.743982    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744039    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744148    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qncb\" (UniqueName: \"kubernetes.io/projected/79b2a99d-951c-4703-b0ee-a163f69b7393-kube-api-access-2qncb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744182    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744202    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744227    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744349    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744389    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.744485 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.744427    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2a99d-951c-4703-b0ee-a163f69b7393-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.845894 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845806    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.845894 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845855    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.845894 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845891    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qncb\" (UniqueName: \"kubernetes.io/projected/79b2a99d-951c-4703-b0ee-a163f69b7393-kube-api-access-2qncb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846201 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845936    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846201 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845958    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846201 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845976    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846201 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.845996    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846201 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846028    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846201 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846058    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2a99d-951c-4703-b0ee-a163f69b7393-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846523 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846270    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846523 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846365    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846523 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846486    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846650 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846543    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.846771 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.846753    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2a99d-951c-4703-b0ee-a163f69b7393-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.848451 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.848429    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.848677 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.848660    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.863277 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.863249    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2a99d-951c-4703-b0ee-a163f69b7393-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.863372 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.863320    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qncb\" (UniqueName: \"kubernetes.io/projected/79b2a99d-951c-4703-b0ee-a163f69b7393-kube-api-access-2qncb\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nw2zt\" (UID: \"79b2a99d-951c-4703-b0ee-a163f69b7393\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:19.913086 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:19.913063    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:20.217957 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:20.217932    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"]
Oct 29 23:06:20.220207 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:06:20.220179    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b2a99d_951c_4703_b0ee_a163f69b7393.slice/crio-66280fbe3050a7a173835f32285c4abf6fe31926cde88675bf74e2504fb1f7db WatchSource:0}: Error finding container 66280fbe3050a7a173835f32285c4abf6fe31926cde88675bf74e2504fb1f7db: Status 404 returned error can't find the container with id 66280fbe3050a7a173835f32285c4abf6fe31926cde88675bf74e2504fb1f7db
Oct 29 23:06:20.222373 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:20.222341    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:06:20.222447 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:20.222407    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:06:20.222447 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:20.222432    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:06:20.934469 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:20.934431    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" event={"ID":"79b2a99d-951c-4703-b0ee-a163f69b7393","Type":"ContainerStarted","Data":"a7ceab2d577f33ea625b4275170263c454d48549418e7e5020c57ade7475d031"}
Oct 29 23:06:20.934892 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:20.934513    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" event={"ID":"79b2a99d-951c-4703-b0ee-a163f69b7393","Type":"ContainerStarted","Data":"66280fbe3050a7a173835f32285c4abf6fe31926cde88675bf74e2504fb1f7db"}
Oct 29 23:06:21.616789 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:21.616744    4451 scope.go:122] "RemoveContainer" containerID="12b08a423e72c0c9a6b05891f4ef53819abf95291d28b2c024bab15bb7959ea7"
Oct 29 23:06:21.915153 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:21.915060    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:21.917127 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:21.917097    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 29 23:06:21.917266 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:21.917155    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 29 23:06:22.917423 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:22.917379    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 29 23:06:22.917976 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:22.917451    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 29 23:06:23.915459 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:23.915408    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 29 23:06:23.915732 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:23.915485    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 29 23:06:24.915419 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:24.915376    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 29 23:06:24.915827 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:24.915461    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 29 23:06:25.915422 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:25.915378    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 29 23:06:25.916055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:25.916008    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 29 23:06:26.914292 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:26.914249    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 29 23:06:26.914473 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:26.914319    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 29 23:06:28.476071 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:28.475852    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:28.476595 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:28.476350    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:28.477306 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:28.477276    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt"
Oct 29 23:06:28.527437 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:06:28.527384    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podStartSLOduration=9.527370701 podStartE2EDuration="9.527370701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:20.998185467 +0000 UTC m=+3120.941144173" watchObservedRunningTime="2026-10-29 23:06:28.527370701 +0000 UTC m=+3128.470329456"
Oct 29 23:06:48.740626 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:06:48.740578    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:07:18.740869 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:07:18.740826    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:07:19.756239 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.756201    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"]
Oct 29 23:07:19.760864 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.760841    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.782211 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.782169    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"]
Oct 29 23:07:19.796320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796100    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nshq\" (UniqueName: \"kubernetes.io/projected/aab2ec53-ffc3-405a-867e-e00373a329cb-kube-api-access-8nshq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796155    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796176    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796202    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab2ec53-ffc3-405a-867e-e00373a329cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796231    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796320 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796296    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796717 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796366    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796717 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796508    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.796717 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.796560    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898077 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898039    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898077 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898077    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898287 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898108    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898287 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898223    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898381 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898355    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nshq\" (UniqueName: \"kubernetes.io/projected/aab2ec53-ffc3-405a-867e-e00373a329cb-kube-api-access-8nshq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898431 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898416    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898494 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898445    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898494 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898487    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab2ec53-ffc3-405a-867e-e00373a329cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898601 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898521    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898601 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898527    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898794 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898766    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898963 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898827    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.898963 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.898871    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.899312 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.899290    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab2ec53-ffc3-405a-867e-e00373a329cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.900616 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.900597    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.900994 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.900972    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.915760 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.915733    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab2ec53-ffc3-405a-867e-e00373a329cb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:19.916101 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:19.916082    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nshq\" (UniqueName: \"kubernetes.io/projected/aab2ec53-ffc3-405a-867e-e00373a329cb-kube-api-access-8nshq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dn5qq\" (UID: \"aab2ec53-ffc3-405a-867e-e00373a329cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:20.077106 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:20.077074    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:20.388973 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:20.388944    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"]
Oct 29 23:07:20.390787 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:07:20.390757    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab2ec53_ffc3_405a_867e_e00373a329cb.slice/crio-3eb592f121e5e13e420d32aa9eef712b6489d41eae09f3c3556d190c7807d99c WatchSource:0}: Error finding container 3eb592f121e5e13e420d32aa9eef712b6489d41eae09f3c3556d190c7807d99c: Status 404 returned error can't find the container with id 3eb592f121e5e13e420d32aa9eef712b6489d41eae09f3c3556d190c7807d99c
Oct 29 23:07:20.392966 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:20.392933    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:07:20.393035 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:20.392998    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:07:20.393035 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:20.393026    4451 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:07:21.219095 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:21.219051    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" event={"ID":"aab2ec53-ffc3-405a-867e-e00373a329cb","Type":"ContainerStarted","Data":"fdcc4b3e1c9429113690fa954f4951e0d67219c36e1745fb3568112f30a98583"}
Oct 29 23:07:21.219551 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:21.219112    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" event={"ID":"aab2ec53-ffc3-405a-867e-e00373a329cb","Type":"ContainerStarted","Data":"3eb592f121e5e13e420d32aa9eef712b6489d41eae09f3c3556d190c7807d99c"}
Oct 29 23:07:21.266592 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:21.266541    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podStartSLOduration=2.266527154 podStartE2EDuration="2.266527154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:21.265077052 +0000 UTC m=+3181.208035759" watchObservedRunningTime="2026-10-29 23:07:21.266527154 +0000 UTC m=+3181.209485906"
Oct 29 23:07:22.077907 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:22.077866    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:22.079433 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:22.079405    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 29 23:07:22.079572 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:22.079465    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 29 23:07:23.078176 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:23.078136    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 29 23:07:23.078583 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:23.078202    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 29 23:07:24.078295 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:24.078255    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 29 23:07:24.078713 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:24.078329    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 29 23:07:25.078059 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:25.078022    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 29 23:07:25.078248 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:25.078077    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 29 23:07:26.078251 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:26.078214    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 29 23:07:26.078753 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:26.078271    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 29 23:07:27.078348 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:27.078306    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 29 23:07:27.078757 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:27.078377    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 29 23:07:29.078289 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:29.078254    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:29.078785 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:29.078310    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:30.120115 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:30.120072    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:30.120544 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:30.120144    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:31.121056 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:31.121016    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:31.121489 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:31.121091    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:32.162067 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:32.162027    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:32.162547 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:32.162079    4451 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq" podUID="aab2ec53-ffc3-405a-867e-e00373a329cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:32.715396 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:32.715364    4451 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:32.715774 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:32.715748    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:32.716528 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:07:32.716507    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dn5qq"
Oct 29 23:07:48.740679 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:07:48.740639    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:08:18.739896 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:08:18.739852    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:08:48.740158 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:08:48.740120    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:09:18.739646 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:09:18.739609    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:09:48.740291 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:09:48.740252    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:10:18.738988 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:10:18.738955    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:10:48.741437 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:10:48.741344    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:11:18.740525 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:11:18.740478    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:11:48.740574 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:11:48.740535    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:12:18.740329 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:12:18.740285    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:12:48.740944 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:12:48.740883    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:13:18.740174 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:13:18.740129    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:13:48.740142 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:13:48.740098    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:14:18.741730 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:14:18.741690    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:14:48.739338 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:14:48.739299    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:15:18.740889 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:15:18.740852    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:15:48.740490 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:15:48.740452    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:16:18.740410 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:16:18.740374    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:16:48.744950 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:16:48.744826    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:16:54.822303 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:54.822266    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"]
Oct 29 23:16:54.822697 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:54.822502    4451 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf" podUID="c83af32f-c131-47ce-9e0d-48d3cd7391f7" containerName="maas-api" containerID="cri-o://15c5412caae8c3259c48d91ce6b7ead482be5f0703f7b3206eb07f016ea63bf9" gracePeriod=30
Oct 29 23:16:56.038007 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.037966    4451 generic.go:356] "Generic (PLEG): container finished" podID="c83af32f-c131-47ce-9e0d-48d3cd7391f7" containerID="15c5412caae8c3259c48d91ce6b7ead482be5f0703f7b3206eb07f016ea63bf9" exitCode=0
Oct 29 23:16:56.038414 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.038035    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf" event={"ID":"c83af32f-c131-47ce-9e0d-48d3cd7391f7","Type":"ContainerDied","Data":"15c5412caae8c3259c48d91ce6b7ead482be5f0703f7b3206eb07f016ea63bf9"}
Oct 29 23:16:56.082444 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.082415    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 23:16:56.118340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.118215    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca\") pod \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") "
Oct 29 23:16:56.118340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.118278    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls\") pod \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") "
Oct 29 23:16:56.118340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.118352    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs\") pod \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") "
Oct 29 23:16:56.118637 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.118382    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7\" (UniqueName: \"kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7\") pod \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\" (UID: \"c83af32f-c131-47ce-9e0d-48d3cd7391f7\") "
Oct 29 23:16:56.118899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.118870    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca" pod "c83af32f-c131-47ce-9e0d-48d3cd7391f7" (UID: "c83af32f-c131-47ce-9e0d-48d3cd7391f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:16:56.121101 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.121062    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls" pod "c83af32f-c131-47ce-9e0d-48d3cd7391f7" (UID: "c83af32f-c131-47ce-9e0d-48d3cd7391f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:56.121232 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.121186    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7" pod "c83af32f-c131-47ce-9e0d-48d3cd7391f7" (UID: "c83af32f-c131-47ce-9e0d-48d3cd7391f7"). InnerVolumeSpecName "kube-api-access-8lbd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:56.121593 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.121563    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs" pod "c83af32f-c131-47ce-9e0d-48d3cd7391f7" (UID: "c83af32f-c131-47ce-9e0d-48d3cd7391f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:56.219258 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.219224    4451 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83af32f-c131-47ce-9e0d-48d3cd7391f7-openshift-service-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:16:56.219258 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.219253    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-maas-api-tls\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:16:56.219258 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.219263    4451 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83af32f-c131-47ce-9e0d-48d3cd7391f7-metrics-certs\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:16:56.219479 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:56.219271    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lbd7\" (UniqueName: \"kubernetes.io/projected/c83af32f-c131-47ce-9e0d-48d3cd7391f7-kube-api-access-8lbd7\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:16:57.046350 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:57.046312    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf" event={"ID":"c83af32f-c131-47ce-9e0d-48d3cd7391f7","Type":"ContainerDied","Data":"8e5dee8dcc84e44198184719509a6bfd4419d93b15c7b2f99aaa28f991a1d7fa"}
Oct 29 23:16:57.046866 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:57.046371    4451 scope.go:122] "RemoveContainer" containerID="15c5412caae8c3259c48d91ce6b7ead482be5f0703f7b3206eb07f016ea63bf9"
Oct 29 23:16:57.046866 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:57.046327    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"
Oct 29 23:16:57.091406 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:57.091380    4451 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"]
Oct 29 23:16:57.101859 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:57.101835    4451 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gngtf"]
Oct 29 23:16:58.167322 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:16:58.167282    4451 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83af32f-c131-47ce-9e0d-48d3cd7391f7" path="/var/lib/kubelet/pods/c83af32f-c131-47ce-9e0d-48d3cd7391f7/volumes"
Oct 29 23:17:03.036348 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.036314    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"]
Oct 29 23:17:03.037003 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.036983    4451 cpu_manager.go:414] "containerMap: removing container" podUID="c83af32f-c131-47ce-9e0d-48d3cd7391f7" containerName="maas-api"
Oct 29 23:17:03.037055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.037007    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83af32f-c131-47ce-9e0d-48d3cd7391f7" containerName="maas-api"
Oct 29 23:17:03.037148 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.037135    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83af32f-c131-47ce-9e0d-48d3cd7391f7" containerName="maas-api"
Oct 29 23:17:03.040954 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.040932    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.054218 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.054189    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"]
Oct 29 23:17:03.062590 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.062562    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:17:03.062693 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.062608    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:17:03.102813 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.102782    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb277\" (UniqueName: \"kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.102813 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.102817    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.103012 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.102841    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.203751 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.203715    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.203893 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.203860    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb277\" (UniqueName: \"kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.204006 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.203890    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.204546 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.204523    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.206176 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.206149    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.222126 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.222100    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb277\" (UniqueName: \"kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277\") pod \"maas-api-revoke-sub-f03629db-dcdbx\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.354606 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.354577    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:03.657388 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:03.657362    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"]
Oct 29 23:17:03.659988 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:17:03.659959    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0e11a2_9106_4251_a59e_88749a893428.slice/crio-ab96c6eeedea8be5f1b4d478cf9f02bfb649abaf05bf632eb40150f693bfd00a WatchSource:0}: Error finding container ab96c6eeedea8be5f1b4d478cf9f02bfb649abaf05bf632eb40150f693bfd00a: Status 404 returned error can't find the container with id ab96c6eeedea8be5f1b4d478cf9f02bfb649abaf05bf632eb40150f693bfd00a
Oct 29 23:17:04.094651 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:04.094616    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx" event={"ID":"fe0e11a2-9106-4251-a59e-88749a893428","Type":"ContainerStarted","Data":"9b89c1c84bbc70ea931175cbe23dcf49fb8fb653a41f351955163c685bbf6fc9"}
Oct 29 23:17:04.095044 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:04.094677    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx" event={"ID":"fe0e11a2-9106-4251-a59e-88749a893428","Type":"ContainerStarted","Data":"ab96c6eeedea8be5f1b4d478cf9f02bfb649abaf05bf632eb40150f693bfd00a"}
Oct 29 23:17:04.136613 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:04.136570    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx" podStartSLOduration=1.136553905 podStartE2EDuration="1.136553905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:04.134610919 +0000 UTC m=+3764.077569623" watchObservedRunningTime="2026-10-29 23:17:04.136553905 +0000 UTC m=+3764.079512609"
Oct 29 23:17:05.100591 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:05.100558    4451 generic.go:356] "Generic (PLEG): container finished" podID="fe0e11a2-9106-4251-a59e-88749a893428" containerID="9b89c1c84bbc70ea931175cbe23dcf49fb8fb653a41f351955163c685bbf6fc9" exitCode=0
Oct 29 23:17:05.100990 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:05.100632    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx" event={"ID":"fe0e11a2-9106-4251-a59e-88749a893428","Type":"ContainerDied","Data":"9b89c1c84bbc70ea931175cbe23dcf49fb8fb653a41f351955163c685bbf6fc9"}
Oct 29 23:17:06.254429 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.254401    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:06.334244 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.334209    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca\") pod \"fe0e11a2-9106-4251-a59e-88749a893428\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") "
Oct 29 23:17:06.334437 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.334305    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277\" (UniqueName: \"kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277\") pod \"fe0e11a2-9106-4251-a59e-88749a893428\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") "
Oct 29 23:17:06.334437 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.334349    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca\") pod \"fe0e11a2-9106-4251-a59e-88749a893428\" (UID: \"fe0e11a2-9106-4251-a59e-88749a893428\") "
Oct 29 23:17:06.334737 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.334704    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca" pod "fe0e11a2-9106-4251-a59e-88749a893428" (UID: "fe0e11a2-9106-4251-a59e-88749a893428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:17:06.336469 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.336444    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca" pod "fe0e11a2-9106-4251-a59e-88749a893428" (UID: "fe0e11a2-9106-4251-a59e-88749a893428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:17:06.336469 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.336453    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277" pod "fe0e11a2-9106-4251-a59e-88749a893428" (UID: "fe0e11a2-9106-4251-a59e-88749a893428"). InnerVolumeSpecName "kube-api-access-wb277". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:17:06.435573 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.435487    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-xks-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:06.435573 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.435520    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb277\" (UniqueName: \"kubernetes.io/projected/fe0e11a2-9106-4251-a59e-88749a893428-kube-api-access-wb277\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:06.435573 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:06.435530    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0e11a2-9106-4251-a59e-88749a893428-maas-api-openshift-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:07.112146 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:07.112115    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx"
Oct 29 23:17:07.112340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:07.112118    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcdbx" event={"ID":"fe0e11a2-9106-4251-a59e-88749a893428","Type":"ContainerDied","Data":"ab96c6eeedea8be5f1b4d478cf9f02bfb649abaf05bf632eb40150f693bfd00a"}
Oct 29 23:17:07.112340 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:07.112268    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab96c6eeedea8be5f1b4d478cf9f02bfb649abaf05bf632eb40150f693bfd00a"
Oct 29 23:17:14.526135 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:14.526054    4451 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:17:14.526793 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:14.526733    4451 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nw2zt" podUID="79b2a99d-951c-4703-b0ee-a163f69b7393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:15.448371 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.448332    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"]
Oct 29 23:17:15.448725 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.448713    4451 cpu_manager.go:414] "containerMap: removing container" podUID="fe0e11a2-9106-4251-a59e-88749a893428" containerName="revoke-keys"
Oct 29 23:17:15.448768 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.448726    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0e11a2-9106-4251-a59e-88749a893428" containerName="revoke-keys"
Oct 29 23:17:15.448819 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.448811    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0e11a2-9106-4251-a59e-88749a893428" containerName="revoke-keys"
Oct 29 23:17:15.452343 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.452320    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.470297 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.470272    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"]
Oct 29 23:17:15.478843 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.478824    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:17:15.478973 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.478957    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:17:15.574837 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.574803    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.575248 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.574843    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.575248 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.574945    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rv9\" (UniqueName: \"kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.675436 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.675408    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rv9\" (UniqueName: \"kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.675536 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.675513    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.675599 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.675545    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.676236 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.676209    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.677938 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.677902    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.692328 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.692296    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rv9\" (UniqueName: \"kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9\") pod \"maas-api-revoke-sub-0165d7d2-25jnq\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:15.765586 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:15.765520    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:16.056248 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:16.056211    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"]
Oct 29 23:17:16.059412 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:17:16.059375    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaebe74_dc8a_441a_8d8d_c8b54d7631b4.slice/crio-2eb3a291b565690554ed28dcd57ddfc135685d2cf0375f98ce90453ad9fa9eec WatchSource:0}: Error finding container 2eb3a291b565690554ed28dcd57ddfc135685d2cf0375f98ce90453ad9fa9eec: Status 404 returned error can't find the container with id 2eb3a291b565690554ed28dcd57ddfc135685d2cf0375f98ce90453ad9fa9eec
Oct 29 23:17:16.162197 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:16.162162    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq" event={"ID":"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4","Type":"ContainerStarted","Data":"2eb3a291b565690554ed28dcd57ddfc135685d2cf0375f98ce90453ad9fa9eec"}
Oct 29 23:17:17.169092 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:17.169009    4451 generic.go:356] "Generic (PLEG): container finished" podID="cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" containerID="d7b9cfda08cc3b7082c9c3d78b840b18f3f01e225f6defcad7e8d1acfec4301c" exitCode=0
Oct 29 23:17:17.169479 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:17.169100    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq" event={"ID":"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4","Type":"ContainerDied","Data":"d7b9cfda08cc3b7082c9c3d78b840b18f3f01e225f6defcad7e8d1acfec4301c"}
Oct 29 23:17:18.305689 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.305665    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:18.401433 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.401406    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca\") pod \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") "
Oct 29 23:17:18.401602 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.401442    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca\") pod \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") "
Oct 29 23:17:18.401602 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.401466    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9\" (UniqueName: \"kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9\") pod \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\" (UID: \"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4\") "
Oct 29 23:17:18.401841 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.401816    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca" pod "cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" (UID: "cbaebe74-dc8a-441a-8d8d-c8b54d7631b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:17:18.403600 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.403572    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9" pod "cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" (UID: "cbaebe74-dc8a-441a-8d8d-c8b54d7631b4"). InnerVolumeSpecName "kube-api-access-68rv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:17:18.403600 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.403589    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca" pod "cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" (UID: "cbaebe74-dc8a-441a-8d8d-c8b54d7631b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:17:18.502382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.502303    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-xks-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:18.502382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.502334    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-maas-api-openshift-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:18.502382 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:18.502348    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rv9\" (UniqueName: \"kubernetes.io/projected/cbaebe74-dc8a-441a-8d8d-c8b54d7631b4-kube-api-access-68rv9\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:18.739981 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:17:18.739942    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:17:19.179472 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:19.179443    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq"
Oct 29 23:17:19.179472 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:19.179452    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-25jnq" event={"ID":"cbaebe74-dc8a-441a-8d8d-c8b54d7631b4","Type":"ContainerDied","Data":"2eb3a291b565690554ed28dcd57ddfc135685d2cf0375f98ce90453ad9fa9eec"}
Oct 29 23:17:19.179710 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:19.179498    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb3a291b565690554ed28dcd57ddfc135685d2cf0375f98ce90453ad9fa9eec"
Oct 29 23:17:26.371408 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.371371    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"]
Oct 29 23:17:26.371800 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.371786    4451 cpu_manager.go:414] "containerMap: removing container" podUID="cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" containerName="revoke-keys"
Oct 29 23:17:26.371859 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.371801    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" containerName="revoke-keys"
Oct 29 23:17:26.371892 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.371887    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaebe74-dc8a-441a-8d8d-c8b54d7631b4" containerName="revoke-keys"
Oct 29 23:17:26.376442 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.376424    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.390261 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.390232    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:17:26.392984 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.392959    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"]
Oct 29 23:17:26.400793 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.400765    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:17:26.476330 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.476299    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6zw\" (UniqueName: \"kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.476535 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.476427    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.476535 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.476485    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.578067 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.578025    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6zw\" (UniqueName: \"kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.578271 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.578125    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.578271 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.578165    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.578865 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.578837    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.580595 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.580572    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.595303 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.595275    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6zw\" (UniqueName: \"kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw\") pod \"maas-api-revoke-sub-6b19338f-v9cmr\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.689848 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.689752    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:26.981384 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:26.981355    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"]
Oct 29 23:17:26.983974 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:17:26.983936    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c000d5_46aa_4251_97be_8224cd5215e0.slice/crio-544b0e33320c171ccc48ae332dddb2e1c3c2f225c105e8b591f244d06e6abdd0 WatchSource:0}: Error finding container 544b0e33320c171ccc48ae332dddb2e1c3c2f225c105e8b591f244d06e6abdd0: Status 404 returned error can't find the container with id 544b0e33320c171ccc48ae332dddb2e1c3c2f225c105e8b591f244d06e6abdd0
Oct 29 23:17:27.215461 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:27.215431    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr" event={"ID":"06c000d5-46aa-4251-97be-8224cd5215e0","Type":"ContainerStarted","Data":"544b0e33320c171ccc48ae332dddb2e1c3c2f225c105e8b591f244d06e6abdd0"}
Oct 29 23:17:28.220985 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:28.220947    4451 generic.go:356] "Generic (PLEG): container finished" podID="06c000d5-46aa-4251-97be-8224cd5215e0" containerID="b5d70d51ef2eecc46611229b84ceac53fa677915fc0077f14629e4733b14d236" exitCode=0
Oct 29 23:17:28.221374 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:28.221030    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr" event={"ID":"06c000d5-46aa-4251-97be-8224cd5215e0","Type":"ContainerDied","Data":"b5d70d51ef2eecc46611229b84ceac53fa677915fc0077f14629e4733b14d236"}
Oct 29 23:17:29.355403 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.355375    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:29.403536 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.403503    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca\") pod \"06c000d5-46aa-4251-97be-8224cd5215e0\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") "
Oct 29 23:17:29.403704 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.403579    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw\" (UniqueName: \"kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw\") pod \"06c000d5-46aa-4251-97be-8224cd5215e0\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") "
Oct 29 23:17:29.403789 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.403756    4451 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca\") pod \"06c000d5-46aa-4251-97be-8224cd5215e0\" (UID: \"06c000d5-46aa-4251-97be-8224cd5215e0\") "
Oct 29 23:17:29.404017 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.403986    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca" pod "06c000d5-46aa-4251-97be-8224cd5215e0" (UID: "06c000d5-46aa-4251-97be-8224cd5215e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:17:29.404155 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.404138    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-openshift-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:29.405832 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.405803    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw" pod "06c000d5-46aa-4251-97be-8224cd5215e0" (UID: "06c000d5-46aa-4251-97be-8224cd5215e0"). InnerVolumeSpecName "kube-api-access-5t6zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:17:29.405949 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.405906    4451 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca" pod "06c000d5-46aa-4251-97be-8224cd5215e0" (UID: "06c000d5-46aa-4251-97be-8224cd5215e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:17:29.504853 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.504774    4451 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t6zw\" (UniqueName: \"kubernetes.io/projected/06c000d5-46aa-4251-97be-8224cd5215e0-kube-api-access-5t6zw\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:29.504853 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:29.504802    4451 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c000d5-46aa-4251-97be-8224cd5215e0-maas-api-xks-ca\") on node \"ip-10-0-143-15.ec2.internal\" DevicePath \"\""
Oct 29 23:17:30.230225 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:30.230201    4451 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr"
Oct 29 23:17:30.230351 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:30.230232    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v9cmr" event={"ID":"06c000d5-46aa-4251-97be-8224cd5215e0","Type":"ContainerDied","Data":"544b0e33320c171ccc48ae332dddb2e1c3c2f225c105e8b591f244d06e6abdd0"}
Oct 29 23:17:30.230351 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:30.230275    4451 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="544b0e33320c171ccc48ae332dddb2e1c3c2f225c105e8b591f244d06e6abdd0"
Oct 29 23:17:48.255513 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:48.255483    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-w6xdh_78d3e1ef-babd-4101-a019-853ecab8fe11/manager/0.log"
Oct 29 23:17:48.443108 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:48.443047    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-69gvh_c4445af6-ecf4-4b6a-b193-1595743cdcc1/manager/0.log"
Oct 29 23:17:48.668589 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:48.668551    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cbglz_99aca75a-d414-4403-b0ef-1a42ec9bf04d/manager/0.log"
Oct 29 23:17:48.740412 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:17:48.740373    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:17:49.269687 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:49.269657    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-45d6d_66f78ee2-8da0-4ff3-87f0-25782d169c28/server/0.log"
Oct 29 23:17:49.455953 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:49.455921    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-v8zwm_52684f1d-5db2-4f57-8002-aca7207fb2d4/manager/0.log"
Oct 29 23:17:49.660060 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:49.660029    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-nzvk9_28ea7ab6-5c97-43db-ba3f-d3ea6552337d/manager/0.log"
Oct 29 23:17:49.860095 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:49.860065    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-dsx95_55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa/copy-manifests/0.log"
Oct 29 23:17:49.897697 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:49.897670    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-dsx95_55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa/manager/0.log"
Oct 29 23:17:50.576384 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:50.576351    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-cvxs6_d3912e35-99c2-41d6-bae8-f4b0d0d2742a/proxy/0.log"
Oct 29 23:17:51.709345 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:51.709310    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-25jnq_cbaebe74-dc8a-441a-8d8d-c8b54d7631b4/revoke-keys/0.log"
Oct 29 23:17:52.260142 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:52.260114    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v9cmr_06c000d5-46aa-4251-97be-8224cd5215e0/revoke-keys/0.log"
Oct 29 23:17:53.145548 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:53.145509    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dcdbx_fe0e11a2-9106-4251-a59e-88749a893428/revoke-keys/0.log"
Oct 29 23:17:53.328096 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:53.328069    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-blj7f_3783398a-165b-4c51-8af9-2ba445dada6b/postgres/0.log"
Oct 29 23:17:56.882476 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:56.882393    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57gktwl_2de4b750-b256-4828-b1a9-d5a5d919c636/istio-proxy/0.log"
Oct 29 23:17:57.619465 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:57.619428    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8d8sf_bb95abec-ee76-4a70-9af9-49ec145c1150/istio-proxy/1.log"
Oct 29 23:17:58.175536 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:58.175507    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-dn5qq_aab2ec53-ffc3-405a-867e-e00373a329cb/istio-proxy/0.log"
Oct 29 23:17:58.727744 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:17:58.727713    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-nw2zt_79b2a99d-951c-4703-b0ee-a163f69b7393/istio-proxy/0.log"
Oct 29 23:18:00.302897 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:00.302866    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8d8sf_bb95abec-ee76-4a70-9af9-49ec145c1150/istio-proxy/0.log"
Oct 29 23:18:11.873304 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:11.873226    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-w6xdh_78d3e1ef-babd-4101-a019-853ecab8fe11/manager/0.log"
Oct 29 23:18:12.071513 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:12.071478    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-69gvh_c4445af6-ecf4-4b6a-b193-1595743cdcc1/manager/0.log"
Oct 29 23:18:12.275149 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:12.275065    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cbglz_99aca75a-d414-4403-b0ef-1a42ec9bf04d/manager/0.log"
Oct 29 23:18:12.701503 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:12.701475    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-45d6d_66f78ee2-8da0-4ff3-87f0-25782d169c28/server/0.log"
Oct 29 23:18:12.900171 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:12.900134    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-v8zwm_52684f1d-5db2-4f57-8002-aca7207fb2d4/manager/0.log"
Oct 29 23:18:13.113700 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:13.113671    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-nzvk9_28ea7ab6-5c97-43db-ba3f-d3ea6552337d/manager/0.log"
Oct 29 23:18:13.329266 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:13.329239    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-dsx95_55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa/copy-manifests/0.log"
Oct 29 23:18:13.367672 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:13.367604    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-dsx95_55de3d00-4ac2-45b3-ada2-d4e0c09bb5fa/manager/0.log"
Oct 29 23:18:14.003690 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:14.003656    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-cvxs6_d3912e35-99c2-41d6-bae8-f4b0d0d2742a/proxy/0.log"
Oct 29 23:18:14.909275 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:14.909234    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-25jnq_cbaebe74-dc8a-441a-8d8d-c8b54d7631b4/revoke-keys/0.log"
Oct 29 23:18:15.456395 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:15.456361    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v9cmr_06c000d5-46aa-4251-97be-8224cd5215e0/revoke-keys/0.log"
Oct 29 23:18:16.317574 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:16.317538    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dcdbx_fe0e11a2-9106-4251-a59e-88749a893428/revoke-keys/0.log"
Oct 29 23:18:16.493995 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:16.493962    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-blj7f_3783398a-165b-4c51-8af9-2ba445dada6b/postgres/0.log"
Oct 29 23:18:18.740672 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:18:18.740638    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:18:19.390448 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:19.390416    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57gktwl_2de4b750-b256-4828-b1a9-d5a5d919c636/istio-proxy/0.log"
Oct 29 23:18:20.150677 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:20.150647    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8d8sf_bb95abec-ee76-4a70-9af9-49ec145c1150/istio-proxy/1.log"
Oct 29 23:18:20.751296 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:20.751260    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-dn5qq_aab2ec53-ffc3-405a-867e-e00373a329cb/istio-proxy/0.log"
Oct 29 23:18:21.363584 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:18:21.363558    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-nw2zt_79b2a99d-951c-4703-b0ee-a163f69b7393/istio-proxy/0.log"
Oct 29 23:18:48.740753 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:18:48.740708    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:19:18.740336 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:19:18.740295    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:19:40.614372 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.614283    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mglbp/must-gather-zvtwt"]
Oct 29 23:19:40.614857 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.614692    4451 cpu_manager.go:414] "containerMap: removing container" podUID="06c000d5-46aa-4251-97be-8224cd5215e0" containerName="revoke-keys"
Oct 29 23:19:40.614857 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.614704    4451 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c000d5-46aa-4251-97be-8224cd5215e0" containerName="revoke-keys"
Oct 29 23:19:40.614857 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.614789    4451 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c000d5-46aa-4251-97be-8224cd5215e0" containerName="revoke-keys"
Oct 29 23:19:40.617980 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.617963    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.630783 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.630762    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mglbp\"/\"kube-root-ca.crt\""
Oct 29 23:19:40.631083 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.631061    4451 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mglbp\"/\"openshift-service-ca.crt\""
Oct 29 23:19:40.631136 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.631061    4451 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mglbp\"/\"default-dockercfg-d8vtb\""
Oct 29 23:19:40.643221 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.643197    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/must-gather-zvtwt"]
Oct 29 23:19:40.728776 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.728744    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa1aed03-3b5c-4659-972d-4c4d7d01a920-must-gather-output\") pod \"must-gather-zvtwt\" (UID: \"fa1aed03-3b5c-4659-972d-4c4d7d01a920\") " pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.728980 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.728812    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/fa1aed03-3b5c-4659-972d-4c4d7d01a920-kube-api-access-c752b\") pod \"must-gather-zvtwt\" (UID: \"fa1aed03-3b5c-4659-972d-4c4d7d01a920\") " pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.830188 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.830145    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa1aed03-3b5c-4659-972d-4c4d7d01a920-must-gather-output\") pod \"must-gather-zvtwt\" (UID: \"fa1aed03-3b5c-4659-972d-4c4d7d01a920\") " pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.830357 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.830225    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/fa1aed03-3b5c-4659-972d-4c4d7d01a920-kube-api-access-c752b\") pod \"must-gather-zvtwt\" (UID: \"fa1aed03-3b5c-4659-972d-4c4d7d01a920\") " pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.830482 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.830460    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa1aed03-3b5c-4659-972d-4c4d7d01a920-must-gather-output\") pod \"must-gather-zvtwt\" (UID: \"fa1aed03-3b5c-4659-972d-4c4d7d01a920\") " pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.853504 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.853472    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/fa1aed03-3b5c-4659-972d-4c4d7d01a920-kube-api-access-c752b\") pod \"must-gather-zvtwt\" (UID: \"fa1aed03-3b5c-4659-972d-4c4d7d01a920\") " pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:40.944858 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:40.944765    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/must-gather-zvtwt"
Oct 29 23:19:41.237453 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:41.237427    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/must-gather-zvtwt"]
Oct 29 23:19:41.240076 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:19:41.240044    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa1aed03_3b5c_4659_972d_4c4d7d01a920.slice/crio-d42427395b0189f7cf043eec68d4ada55a81120f1ff73c4746eb603983607389 WatchSource:0}: Error finding container d42427395b0189f7cf043eec68d4ada55a81120f1ff73c4746eb603983607389: Status 404 returned error can't find the container with id d42427395b0189f7cf043eec68d4ada55a81120f1ff73c4746eb603983607389
Oct 29 23:19:41.241885 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:41.241869    4451 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:19:41.826473 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:41.826433    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/must-gather-zvtwt" event={"ID":"fa1aed03-3b5c-4659-972d-4c4d7d01a920","Type":"ContainerStarted","Data":"d42427395b0189f7cf043eec68d4ada55a81120f1ff73c4746eb603983607389"}
Oct 29 23:19:42.832842 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:42.832804    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/must-gather-zvtwt" event={"ID":"fa1aed03-3b5c-4659-972d-4c4d7d01a920","Type":"ContainerStarted","Data":"4464b48ca29c7543a5741b1c381b96bd3a6817ed2ea10fe528cc8b5d3a4cae42"}
Oct 29 23:19:42.833257 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:42.832853    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/must-gather-zvtwt" event={"ID":"fa1aed03-3b5c-4659-972d-4c4d7d01a920","Type":"ContainerStarted","Data":"cfe053262dac5cd4bdb164cd1a45aabce087102e9e5ac83e99e18e7490e200b5"}
Oct 29 23:19:42.867812 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:42.867754    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mglbp/must-gather-zvtwt" podStartSLOduration=2.066859639 podStartE2EDuration="2.86774103s" totalImagesPullingTime="800.881391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:42.867073229 +0000 UTC m=+3922.810031935" watchObservedRunningTime="2026-10-29 23:19:42.86774103 +0000 UTC m=+3922.810699735"
Oct 29 23:19:45.105124 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:45.105066    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j9lz9_bbeed871-241a-4509-ac94-dbe5170904fe/global-pull-secret-syncer/0.log"
Oct 29 23:19:46.253658 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:46.253622    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x4bzm_3e49e136-156b-4dfa-a877-51b5c57d0ffa/konnectivity-agent/0.log"
Oct 29 23:19:46.619585 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:46.619551    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-15.ec2.internal_b64de3cb655c6c434e0a94257c87f3e3/haproxy/0.log"
Oct 29 23:19:48.801932 ip-10-0-143-15 kubenswrapper[4451]: E1029 23:19:48.799224    4451 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qvpbn" containerName="extractor"
Oct 29 23:19:55.020103 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.020052    4451 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"]
Oct 29 23:19:55.028175 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.028153    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.051665 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.051635    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"]
Oct 29 23:19:55.207557 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.207516    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-lib-modules\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.207727 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.207572    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-proc\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.207727 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.207673    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-sys\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.207727 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.207724    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-podres\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.207888 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.207833    4451 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hl9k\" (UniqueName: \"kubernetes.io/projected/bfb93957-baa2-42b2-a955-512cd8c14695-kube-api-access-6hl9k\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309558 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309477    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-sys\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309558 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309536    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-podres\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309767 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309588    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hl9k\" (UniqueName: \"kubernetes.io/projected/bfb93957-baa2-42b2-a955-512cd8c14695-kube-api-access-6hl9k\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309767 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309609    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-sys\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309767 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309686    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-lib-modules\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309767 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309709    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-podres\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.309767 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309728    4451 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-proc\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.310037 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309809    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-lib-modules\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.310037 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.309854    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfb93957-baa2-42b2-a955-512cd8c14695-proc\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.327070 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.327040    4451 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hl9k\" (UniqueName: \"kubernetes.io/projected/bfb93957-baa2-42b2-a955-512cd8c14695-kube-api-access-6hl9k\") pod \"perf-node-gather-daemonset-6lbkp\" (UID: \"bfb93957-baa2-42b2-a955-512cd8c14695\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.357197 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.357148    4451 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.658334 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.658298    4451 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"]
Oct 29 23:19:55.664641 ip-10-0-143-15 kubenswrapper[4451]: W1029 23:19:55.664597    4451 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfb93957_baa2_42b2_a955_512cd8c14695.slice/crio-74646106b01e8eef16df9f4d106d0443338a7a7a9280ddd008b0d9db3979e742 WatchSource:0}: Error finding container 74646106b01e8eef16df9f4d106d0443338a7a7a9280ddd008b0d9db3979e742: Status 404 returned error can't find the container with id 74646106b01e8eef16df9f4d106d0443338a7a7a9280ddd008b0d9db3979e742
Oct 29 23:19:55.882516 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.882491    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-27rcx_151d07f9-8797-4795-94b8-4cee326c14ea/kube-state-metrics/0.log"
Oct 29 23:19:55.980047 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.979958    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp" event={"ID":"bfb93957-baa2-42b2-a955-512cd8c14695","Type":"ContainerStarted","Data":"1c9b6212e7a14746d7a162d5883065c472068138d12712233f95b9fe23fa4a63"}
Oct 29 23:19:55.980047 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.980019    4451 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp" event={"ID":"bfb93957-baa2-42b2-a955-512cd8c14695","Type":"ContainerStarted","Data":"74646106b01e8eef16df9f4d106d0443338a7a7a9280ddd008b0d9db3979e742"}
Oct 29 23:19:55.980240 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.980051    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:19:55.985433 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:55.985407    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-27rcx_151d07f9-8797-4795-94b8-4cee326c14ea/kube-rbac-proxy-main/0.log"
Oct 29 23:19:56.016000 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:56.015942    4451 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp" podStartSLOduration=2.015892696 podStartE2EDuration="2.015892696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:56.013494829 +0000 UTC m=+3935.956453535" watchObservedRunningTime="2026-10-29 23:19:56.015892696 +0000 UTC m=+3935.958851402"
Oct 29 23:19:56.101000 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:56.100972    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-27rcx_151d07f9-8797-4795-94b8-4cee326c14ea/kube-rbac-proxy-self/0.log"
Oct 29 23:19:56.337200 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:56.337172    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-76vbk_7b25a6f4-8818-4df3-84d1-0302f82de501/monitoring-plugin/0.log"
Oct 29 23:19:57.154087 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.154058    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qlr9r_bd4f7225-ed09-4c35-bd1a-3559ab9a020c/node-exporter/0.log"
Oct 29 23:19:57.267874 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.267843    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qlr9r_bd4f7225-ed09-4c35-bd1a-3559ab9a020c/kube-rbac-proxy/0.log"
Oct 29 23:19:57.381383 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.381357    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qlr9r_bd4f7225-ed09-4c35-bd1a-3559ab9a020c/init-textfile/0.log"
Oct 29 23:19:57.499933 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.499828    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tnlx5_b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad/kube-rbac-proxy-main/0.log"
Oct 29 23:19:57.613712 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.613664    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tnlx5_b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad/kube-rbac-proxy-self/0.log"
Oct 29 23:19:57.726348 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.726313    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tnlx5_b6d0efc5-1efd-44b2-a4c7-d5c8f42808ad/openshift-state-metrics/0.log"
Oct 29 23:19:57.853630 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.853601    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/prometheus/0.log"
Oct 29 23:19:57.966458 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:57.966429    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/config-reloader/0.log"
Oct 29 23:19:58.079441 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.079411    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/thanos-sidecar/0.log"
Oct 29 23:19:58.194879 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.194802    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/kube-rbac-proxy-web/0.log"
Oct 29 23:19:58.308055 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.308023    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/kube-rbac-proxy/0.log"
Oct 29 23:19:58.421943 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.421898    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/kube-rbac-proxy-thanos/0.log"
Oct 29 23:19:58.535606 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.535511    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a212c3a9-87bf-4ce9-b528-fcfa1076ecde/init-config-reloader/0.log"
Oct 29 23:19:58.662345 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.662312    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r47ld_b29bb251-98b5-49bd-bb0c-bee62b05e783/prometheus-operator/0.log"
Oct 29 23:19:58.767312 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.767280    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r47ld_b29bb251-98b5-49bd-bb0c-bee62b05e783/kube-rbac-proxy/0.log"
Oct 29 23:19:58.883996 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:58.883965    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-f9ndh_19f32a30-891d-46eb-928b-3848afe135d4/prometheus-operator-admission-webhook/0.log"
Oct 29 23:19:59.002806 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:59.002773    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cd9b7f765-g5nwx_67122eae-b9ce-43a7-b0de-691a8ee97a21/telemeter-client/0.log"
Oct 29 23:19:59.115185 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:59.115157    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cd9b7f765-g5nwx_67122eae-b9ce-43a7-b0de-691a8ee97a21/reload/0.log"
Oct 29 23:19:59.229693 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:19:59.229615    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cd9b7f765-g5nwx_67122eae-b9ce-43a7-b0de-691a8ee97a21/kube-rbac-proxy/0.log"
Oct 29 23:20:01.997793 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:01.997764    4451 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-6lbkp"
Oct 29 23:20:03.636899 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:03.636870    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-2gb5b_cd5bf93c-bbbd-478a-9f00-d115f3d97475/download-server/0.log"
Oct 29 23:20:07.201446 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:07.201408    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2lqth_7bb945a9-e793-4e53-bb13-38b3eb5022e4/dns/0.log"
Oct 29 23:20:07.314401 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:07.314367    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2lqth_7bb945a9-e793-4e53-bb13-38b3eb5022e4/kube-rbac-proxy/0.log"
Oct 29 23:20:07.792023 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:07.791994    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57gktwl_2de4b750-b256-4828-b1a9-d5a5d919c636/istio-proxy/0.log"
Oct 29 23:20:08.017453 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:08.017408    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7cws_41bc42bb-f14c-4921-a5f4-5a7ceec653e6/dns-node-resolver/0.log"
Oct 29 23:20:09.346144 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:09.346085    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5q62r_0160f0a0-c1cf-4e77-9ed6-4d2cfd45e054/node-ca/0.log"
Oct 29 23:20:10.580366 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:10.580328    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8d8sf_bb95abec-ee76-4a70-9af9-49ec145c1150/istio-proxy/0.log"
Oct 29 23:20:10.583847 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:10.583820    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8d8sf_bb95abec-ee76-4a70-9af9-49ec145c1150/istio-proxy/1.log"
Oct 29 23:20:10.751922 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:10.751892    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-dn5qq_aab2ec53-ffc3-405a-867e-e00373a329cb/istio-proxy/0.log"
Oct 29 23:20:10.930354 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:10.930269    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-nw2zt_79b2a99d-951c-4703-b0ee-a163f69b7393/istio-proxy/0.log"
Oct 29 23:20:11.443811 ip-10-0-143-15 kubenswrapper[4451]: I1029 23:20:11.443769    4451 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57gktwl_2de4b750-b256-4828-b1a9-d5a5d919c636/istio-proxy/0.log"